Jul 06 23:41:10.601648 ip-10-0-132-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 23:41:10.601662 ip-10-0-132-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 23:41:10.601670 ip-10-0-132-183 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 23:41:10.601927 ip-10-0-132-183 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 23:41:20.144647 ip-10-0-132-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 06 23:41:20.144664 ip-10-0-132-183 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3a31d447127f4ed5b002d90e180bd697 -- Jul 06 23:42:48.354642 ip-10-0-132-183 systemd[1]: Starting Kubernetes Kubelet... Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.817814 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823378 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823394 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823397 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823400 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823404 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823407 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823409 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:42:48.828479 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823413 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823416 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823418 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823421 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823424 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823427 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823430 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823433 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823435 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823443 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823446 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823448 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823452 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823454 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823457 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823463 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823467 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823470 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823473 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:42:48.830574 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823476 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823479 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823482 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823485 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823488 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823491 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823493 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823496 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823498 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823502 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823505 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823508 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823511 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823514 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823518 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823520 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823523 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823526 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823529 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:42:48.832936 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823532 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823535 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823537 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823540 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823543 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823546 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823550 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823552 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823555 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823559 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823561 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823564 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823567 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823570 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823572 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823575 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823577 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823580 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823582 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823585 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:42:48.848209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823588 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823590 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823593 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823596 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823598 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823601 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823604 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823607 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823609 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823612 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823615 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823619 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823622 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823626 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823629 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823631 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823634 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823637 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823639 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823642 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:42:48.850490 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823645 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823647 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823651 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823654 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823657 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823660 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823662 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823664 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823667 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823670 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823673 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823675 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823678 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.823681 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824442 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824450 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824456 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824459 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824462 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824466 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:42:48.852429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824470 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824472 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824475 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824478 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824480 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824484 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824488 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824491 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824494 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824496 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824499 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824502 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824504 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824507 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824509 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824512 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824515 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824517 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824520 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824523 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:42:48.854818 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824525 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824528 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824531 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824534 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824537 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824539 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824542 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824544 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824547 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824550 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824552 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824555 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824558 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824560 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824563 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824565 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824567 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824570 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824572 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824575 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:42:48.868190 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824577 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824580 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824583 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824586 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824591 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824594 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824596 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824599 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824602 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824604 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824607 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824609 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824612 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824615 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824620 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824622 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824625 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824628 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824631 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:42:48.870506 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824633 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824636 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824639 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824643 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824645 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824648 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824651 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824653 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824656 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824659 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824661 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824664 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824667 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824670 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824673 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824675 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824678 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824680 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824683 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824685 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:42:48.872375 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824688 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824690 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824693 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824696 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824698 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824700 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824703 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824708 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824711 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824714 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824716 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824719 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824722 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.824724 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825127 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825132 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825136 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825141 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825145 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825148 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:42:48.874650 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825151 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825154 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825157 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825160 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825163 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825165 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825168 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825170 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825173 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825175 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825178 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825181 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825183 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825186 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825188 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825191 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825193 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825196 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825199 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825202 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:42:48.888148 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825205 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825208 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825210 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825212 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825215 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825219 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825222 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825225 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825227 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825231 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825234 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825237 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825240 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825242 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825247 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825250 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825253 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825256 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825259 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:42:48.890775 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825262 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:42:48.889119 ip-10-0-132-183 systemd[1]: Started Kubernetes Kubelet. Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825264 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825267 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825270 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825272 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825275 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825277 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825280 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825296 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825299 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825301 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825304 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825308 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825310 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825313 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825315 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825318 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825320 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825323 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825325 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:42:48.893111 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825328 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825331 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825335 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825338 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825340 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825343 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825346 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825348 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825351 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825354 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825356 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825359 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825361 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825363 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825366 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825369 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825371 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825374 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825376 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825379 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:42:48.910508 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825381 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825384 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825386 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825389 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825393 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825395 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825399 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825403 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825405 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825408 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825411 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825413 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825416 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.825419 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825500 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825507 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825514 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825518 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825523 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825527 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825531 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 23:42:49.115362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825535 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825539 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825542 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825545 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825548 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825551 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825554 2564 flags.go:64] FLAG: --cgroup-root="" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825558 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825561 2564 flags.go:64] FLAG: --client-ca-file="" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825564 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825566 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825571 2564 flags.go:64] FLAG: --cluster-domain="" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825574 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825577 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825580 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825584 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825588 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825591 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825594 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825597 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825600 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825603 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825606 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825609 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 23:42:49.129183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825612 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825618 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825621 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825624 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825627 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825630 2564 flags.go:64] FLAG: --enable-server="true" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825633 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825638 2564 flags.go:64] FLAG: --event-burst="100" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825641 2564 flags.go:64] FLAG: --event-qps="50" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825644 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825647 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825650 2564 flags.go:64] FLAG: --eviction-hard="" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825654 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825657 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825660 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825663 2564 flags.go:64] FLAG: --eviction-soft="" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825666 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825669 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825671 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825675 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825678 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825681 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825683 2564 flags.go:64] FLAG: --feature-gates="" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825688 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825691 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 23:42:49.145138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825694 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825697 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825700 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825703 2564 flags.go:64] FLAG: --help="false" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825706 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825709 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825712 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825715 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825720 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825723 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825726 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825729 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825732 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825735 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825739 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825742 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825745 2564 flags.go:64] FLAG: --kube-reserved="" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825748 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825751 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825754 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825757 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825760 2564 flags.go:64] FLAG: --lock-file="" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825763 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825766 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 23:42:49.146658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825769 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825774 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825777 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825780 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825782 2564 flags.go:64] FLAG: --logging-format="text" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825785 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825790 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825792 2564 flags.go:64] FLAG: --manifest-url="" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825796 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825800 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825803 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825807 2564 flags.go:64] FLAG: --max-pods="110" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825810 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825813 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825816 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825818 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825822 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825826 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825828 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825836 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825839 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825842 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825845 2564 flags.go:64] FLAG: --pod-cidr="" Jul 06 23:42:49.147795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825849 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825854 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825857 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825861 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825864 2564 flags.go:64] FLAG: --port="10250" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825867 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825870 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a36f40a903f2ecd8" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825874 2564 flags.go:64] FLAG: --qos-reserved="" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825876 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825879 2564 flags.go:64] FLAG: --register-node="true" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825882 2564 flags.go:64] FLAG: --register-with-taints="" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825886 2564 flags.go:64] FLAG: --registry-burst="10" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825889 2564 flags.go:64] FLAG: --registry-qps="5" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825891 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825894 2564 flags.go:64] FLAG: --reserved-memory="" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825898 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825905 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825908 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825911 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825914 2564 flags.go:64] FLAG: --runonce="false" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825917 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825920 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825923 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825926 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825928 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825933 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 23:42:49.148926 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825936 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825939 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825942 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825945 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825948 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825950 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825953 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825957 2564 flags.go:64] FLAG: --system-cgroups="" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825960 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825965 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825967 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825970 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825974 2564 flags.go:64] FLAG: --tls-min-version="" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825977 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825980 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825982 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825985 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825988 2564 flags.go:64] FLAG: --v="2" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825992 2564 flags.go:64] FLAG: --version="false" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.825996 2564 flags.go:64] FLAG: --vmodule="" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.826000 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.826003 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826135 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826139 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:42:49.150064 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826142 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826145 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826148 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826151 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826154 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826157 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826160 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826164 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826167 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826170 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826173 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826175 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826178 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826180 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826183 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826185 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826188 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826191 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826193 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826196 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:42:49.151201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826199 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826201 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826203 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826206 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826208 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826211 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826213 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826216 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826219 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826221 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826225 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826227 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826230 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826232 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826235 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826238 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826240 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826243 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826245 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826249 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826251 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:42:49.152104 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826254 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826256 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826259 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826262 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826264 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826267 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826269 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826272 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826275 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826277 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826300 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826303 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826306 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826309 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826312 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826314 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826317 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826319 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826322 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826325 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:42:49.153264 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826327 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826332 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826335 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826337 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826340 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826342 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826345 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826348 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826350 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826353 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826358 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826361 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826364 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826366 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826370 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826374 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826377 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826380 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826385 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:42:49.154209 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826388 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826392 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826398 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826401 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826404 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826407 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826410 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826413 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826416 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826418 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826421 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826425 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826428 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826431 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826435 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826438 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:42:49.155130 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.826441 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.827135 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.834019 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.834036 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.834058 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.834068 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834131 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834137 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834140 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834144 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834147 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834151 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834154 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834157 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:42:49.155802 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834160 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834164 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834168 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834172 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834176 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834179 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834182 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834185 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834188 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834191 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834194 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834197 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834199 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834203 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834206 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834209 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834212 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834214 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834217 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834220 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:42:49.156482 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834223 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834225 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834228 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834230 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834233 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834236 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834239 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834241 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834244 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834246 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834249 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834251 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834254 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834256 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834259 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834262 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834265 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834268 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834272 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834274 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:42:49.157600 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834277 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834279 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834299 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834302 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834305 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834308 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834310 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834314 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834318 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834321 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834324 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834327 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834329 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834332 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834335 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834337 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834340 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834343 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834346 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834349 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:42:49.158507 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834351 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834354 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834357 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834359 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834363 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834365 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834368 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834388 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834392 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834396 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834402 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834405 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834409 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834413 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834416 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834419 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834422 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834425 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834427 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:42:49.159419 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834430 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834433 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834435 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834438 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834441 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834444 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834446 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834449 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834452 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834454 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834457 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834460 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.834465 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834583 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834588 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:42:49.160334 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834591 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834595 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834599 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834602 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834605 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834609 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834612 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834615 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834618 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834623 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834627 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834630 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834632 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834635 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834638 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834640 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834643 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834645 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834648 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:42:49.161016 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834650 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834653 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834656 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834658 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834661 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834663 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834666 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834669 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834671 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834674 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834676 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834679 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834681 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834684 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834686 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834689 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834691 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834694 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834697 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834700 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:42:49.161950 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834702 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834705 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834710 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834713 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834716 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834719 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834721 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834724 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834726 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834729 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834731 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834734 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834737 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834739 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834742 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834744 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834747 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834749 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834752 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834755 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:42:49.162841 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834757 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834760 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834762 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834765 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834768 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834770 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834773 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834776 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834778 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834781 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834783 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834786 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834789 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834791 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834795 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834798 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834801 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834803 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834806 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834809 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:42:49.163748 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834811 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834813 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834816 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834819 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834821 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834824 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834828 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834831 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834834 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834836 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834839 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834841 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834844 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834846 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834849 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834853 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834856 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:42:49.166978 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:48.834859 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.834864 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.835000 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.837202 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.838606 2564 server.go:1019] "Starting client certificate rotation" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.838712 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.838754 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.841490 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.844635 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.851070 2564 log.go:25] "Validated CRI v1 runtime API" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.853494 2564 log.go:25] "Validated CRI v1 image API" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.854203 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 23:42:49.167882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.856733 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a80c408-99f1-4525-8647-0ca150acd73c:/dev/nvme0n1p4 e414c761-1dc3-404b-b633-07470cea1da8:/dev/nvme0n1p3] Jul 06 23:42:49.398627 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.856749 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.861033 2564 manager.go:216] Machine: {Timestamp:2026-07-06 23:42:48.860543639 +0000 UTC m=+0.371793022 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099766 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22ac0d358a48aa689f8e0f7ca371a8 SystemUUID:ec22ac0d-358a-48aa-689f-8e0f7ca371a8 BootID:3a31d447-127f-4ed5-b002-d90e180bd697 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3a:18:6e:b9:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3a:18:6e:b9:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:2a:a8:dd:44:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.861131 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.861213 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.865442 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.865470 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.866938 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.866951 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 23:42:49.399123 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.866959 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.867768 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.869426 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.870901 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874341 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874479 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874506 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874516 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874530 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874539 2564 kubelet.go:398] "Adding apiserver pod source" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874548 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874955 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.874974 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.877754 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.881491 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.881532 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882871 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882899 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882910 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882918 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882928 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882934 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882944 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882949 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882956 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882963 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882972 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.882978 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.883931 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.883938 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.885415 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.885527 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.885657 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.888163 2564 server.go:1268] "Started kubelet" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.888250 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.888334 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.889012 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.890520 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.891236 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 06 23:42:49.399500 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.894055 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.898246 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.898254 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.898367 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.898925 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.898941 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.898940 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899024 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899046 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899082 2564 factory.go:55] Registering systemd factory Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899131 2564 factory.go:223] Registration of the systemd container factory successfully Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899355 2564 factory.go:153] Registering CRI-O factory Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899370 2564 factory.go:223] Registration of the crio container factory successfully Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899418 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899441 2564 factory.go:103] Registering Raw factory Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.899458 2564 manager.go:1196] Started watching for new ooms in manager Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.900571 2564 manager.go:349] Starting recovery of all containers Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.902595 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.902794 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.902681 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-183.ec2.internal.18bfd7e7ffa09691 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-183.ec2.internal,UID:ip-10-0-132-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-183.ec2.internal,},FirstTimestamp:2026-07-06 23:42:48.888137361 +0000 UTC m=+0.399386745,LastTimestamp:2026-07-06 23:42:48.888137361 +0000 UTC m=+0.399386745,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-183.ec2.internal,}" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.914737 2564 manager.go:354] Recovery completed Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.916490 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-scpdn" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.919072 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.920991 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.921016 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.921027 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.921553 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.921562 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.921577 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.921561 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.922469 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.922501 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.922514 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:42:49.400574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.922895 2564 policy_none.go:49] "None policy: Start" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.922916 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.922930 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.923153 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-183.ec2.internal.18bfd7e801961147 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-183.ec2.internal,UID:ip-10-0-132-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-183.ec2.internal,},FirstTimestamp:2026-07-06 23:42:48.921002311 +0000 UTC m=+0.432251693,LastTimestamp:2026-07-06 23:42:48.921002311 +0000 UTC m=+0.432251693,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-183.ec2.internal,}" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.923849 2564 policy_none.go:47] "Start" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.928351 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-scpdn" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.984862 2564 manager.go:337] "Starting Device Plugin manager" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.984894 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.984905 2564 server.go:87] "Starting device plugin registration server" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.985198 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.985211 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.985367 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.985461 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:48.985475 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.985881 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:48.985935 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.005499 2564 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2599 into the system container \"/system.slice\": " Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.086059 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.087124 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.087151 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.087161 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.087188 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.096146 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.096174 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-183.ec2.internal\": node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.120279 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.162106 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.163643 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.163683 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.163709 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.163748 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.166276 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.220830 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:49.401465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.263995 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal"] Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.264097 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.265149 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.265175 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.265184 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.266436 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.266545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.266573 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.268108 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.268119 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.268141 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.268151 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.268125 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.268204 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.269477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.269498 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.270166 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.270182 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.270192 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.301172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9064f225fd7e0619db91cfef5c8bbc9d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-183.ec2.internal\" (UID: \"9064f225fd7e0619db91cfef5c8bbc9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.301210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d5995c2cad545e25efb8ab86dbae04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c9d5995c2cad545e25efb8ab86dbae04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.301235 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d5995c2cad545e25efb8ab86dbae04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c9d5995c2cad545e25efb8ab86dbae04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.304193 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-183.ec2.internal\" not found" node="ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.307767 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-183.ec2.internal\" not found" node="ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.321742 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:49.402373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.401745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d5995c2cad545e25efb8ab86dbae04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c9d5995c2cad545e25efb8ab86dbae04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.403063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.401777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d5995c2cad545e25efb8ab86dbae04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c9d5995c2cad545e25efb8ab86dbae04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.403063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.401792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9064f225fd7e0619db91cfef5c8bbc9d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-183.ec2.internal\" (UID: \"9064f225fd7e0619db91cfef5c8bbc9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.403063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.401826 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9064f225fd7e0619db91cfef5c8bbc9d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-183.ec2.internal\" (UID: \"9064f225fd7e0619db91cfef5c8bbc9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.403063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.401827 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d5995c2cad545e25efb8ab86dbae04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c9d5995c2cad545e25efb8ab86dbae04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:49.403063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:49.401882 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d5995c2cad545e25efb8ab86dbae04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c9d5995c2cad545e25efb8ab86dbae04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:51.071793 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.421825 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.071793 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:49.522336 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.231688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.091182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:51.231688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.095099 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 06 23:42:51.231688 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:51.107607 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.231688 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:51.208150 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.305889 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:51.308542 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.322705 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.322835 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.322874 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:51.409080 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.414155 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 23:37:50 +0000 UTC" deadline="2028-02-12 05:42:01.225943007 +0000 UTC" Jul 06 23:42:51.460666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.414183 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14045h59m9.811763401s" Jul 06 23:42:51.664392 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:51.509746 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.664392 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:51.610311 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.664392 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:51.653041 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d5995c2cad545e25efb8ab86dbae04.slice/crio-fc56ea846bb53efe8db4a2af291de9f3f89e35f711d0751bda643c73b4c30e3f WatchSource:0}: Error finding container fc56ea846bb53efe8db4a2af291de9f3f89e35f711d0751bda643c73b4c30e3f: Status 404 returned error can't find the container with id fc56ea846bb53efe8db4a2af291de9f3f89e35f711d0751bda643c73b4c30e3f Jul 06 23:42:51.664392 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.657770 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.690905 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:51.693191 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9064f225fd7e0619db91cfef5c8bbc9d.slice/crio-02ad7add0da52467a68b61807a5c0ce4ec21f66bb51222522243ffa88472727f WatchSource:0}: Error finding container 02ad7add0da52467a68b61807a5c0ce4ec21f66bb51222522243ffa88472727f: Status 404 returned error can't find the container with id 02ad7add0da52467a68b61807a5c0ce4ec21f66bb51222522243ffa88472727f Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:51.710760 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.769340 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.783188 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.794763 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.795886 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 06 23:42:51.868969 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:51.803923 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.443542 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.443663 2564 apiserver.go:52] "Watching apiserver" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.450247 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.450687 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal","openshift-multus/multus-24bw4","openshift-multus/multus-additional-cni-plugins-6rf6b","openshift-multus/network-metrics-daemon-6lqdv","kube-system/global-pull-secret-syncer-7rs8n","kube-system/konnectivity-agent-8qrzv","openshift-image-registry/node-ca-sd5wt","openshift-network-diagnostics/network-check-target-jj75g","openshift-network-operator/iptables-alerter-4vcbf","openshift-ovn-kubernetes/ovnkube-node-d9pgh","kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9","openshift-cluster-node-tuning-operator/tuned-ljs8w","openshift-dns/node-resolver-brd86"] Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.452175 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.453007 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.453356 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.454387 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.454603 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.455169 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.454919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xgb5z\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.455659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.455891 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.455967 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.457053 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.457092 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jgwlf\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.457604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.458604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.459291 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.459320 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.459334 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.459499 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sv9ks\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.459782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.459836 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 23:42:52.466870 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.459888 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.461422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.463572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.463990 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.464274 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.464530 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xz4w7\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.465838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.466043 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.466140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.466253 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.466774 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kwfhg\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.467999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.468577 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qpv7j\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.468813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 23:42:52.469309 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.469000 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 23:42:52.470162 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.469706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.473153 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.471503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:52.473153 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.471606 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:42:52.473153 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.472059 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p8dfq\"" Jul 06 23:42:52.473153 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.472107 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 23:42:52.473153 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.472859 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 23:42:52.473153 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.473104 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6qttn\"" Jul 06 23:42:52.473153 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.473143 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.474445 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.473375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 23:42:52.474445 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.473564 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 23:42:52.474445 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.473704 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 23:42:52.474445 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.473892 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 23:42:52.474445 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.474099 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 23:42:52.482693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.475753 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r27xx" Jul 06 23:42:52.482693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.475827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 23:42:52.482693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.476015 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 23:42:52.482693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.476342 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 23:42:52.482693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.476614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k47kl\"" Jul 06 23:42:52.485455 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.484449 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 23:42:52.487139 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.486604 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r27xx" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2nb\" (UniqueName: \"kubernetes.io/projected/670307dd-877f-4f25-ad8c-3b6bb6ce1134-kube-api-access-5q2nb\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-kubelet-config\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj27c\" (UniqueName: \"kubernetes.io/projected/c4977763-ab27-4d4d-b91a-28e2dac29776-kube-api-access-xj27c\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-system-cni-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-var-lib-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-k8s-cni-cncf-io\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysctl-d\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovnkube-config\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4977763-ab27-4d4d-b91a-28e2dac29776-iptables-alerter-script\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.543978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc12b8c-a240-473d-a57c-b9cd825797ef-tmp-dir\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-multus-certs\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.544042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysctl-conf\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-etc-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwz6z\" (UniqueName: \"kubernetes.io/projected/ceff6ec7-0ad5-413a-b36b-124caf4231af-kube-api-access-lwz6z\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-cni-multus\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-hostroot\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrhn\" (UniqueName: \"kubernetes.io/projected/52506102-c9ef-4fd1-b154-19c1a64604f3-kube-api-access-krrhn\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-os-release\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-netns\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.547163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-dbus\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-cnibin\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cnibin\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-env-overrides\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4977763-ab27-4d4d-b91a-28e2dac29776-host-slash\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-ovn\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-node-log\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544786 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-systemd\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-host\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-slash\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgrz\" (UniqueName: \"kubernetes.io/projected/5b169c47-57aa-49ee-ba53-7127a0795135-kube-api-access-xpgrz\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bc12b8c-a240-473d-a57c-b9cd825797ef-hosts-file\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.548020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.544938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-socket-dir-parent\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-sys\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-var-lib-kubelet\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-cni-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-conf-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-run\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-lib-modules\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-tuned\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-sys-fs\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-serviceca\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsrv\" (UniqueName: \"kubernetes.io/projected/9bc12b8c-a240-473d-a57c-b9cd825797ef-kube-api-access-9jsrv\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52506102-c9ef-4fd1-b154-19c1a64604f3-cni-binary-copy\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/478986e2-196e-4b09-8536-4f2d903d6c39-agent-certs\") pod \"konnectivity-agent-8qrzv\" (UID: \"478986e2-196e-4b09-8536-4f2d903d6c39\") " pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-kubelet\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.554756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545383 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-socket-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-registration-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-os-release\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-etc-kubernetes\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-systemd-units\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-cni-bin\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-system-cni-dir\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-log-socket\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-run-ovn-kubernetes\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-cni-netd\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovn-node-metrics-cert\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-kubelet\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-host\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.556327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfq6h\" (UniqueName: \"kubernetes.io/projected/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-kube-api-access-bfq6h\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/478986e2-196e-4b09-8536-4f2d903d6c39-konnectivity-ca\") pod \"konnectivity-agent-8qrzv\" (UID: \"478986e2-196e-4b09-8536-4f2d903d6c39\") " pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-systemd\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacbedcb-b268-4879-a37b-8dc034a11e65-tmp\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lktb\" (UniqueName: \"kubernetes.io/projected/cacbedcb-b268-4879-a37b-8dc034a11e65-kube-api-access-5lktb\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.545947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-cni-bin\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysconfig\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-kubernetes\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovnkube-script-lib\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546235 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-device-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-etc-selinux\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-daemon-config\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.558688 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-modprobe-d\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.559591 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6ws\" (UniqueName: \"kubernetes.io/projected/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-kube-api-access-xc6ws\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.559591 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.546391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-run-netns\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.626309 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-os-release\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-etc-kubernetes\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-systemd-units\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-cni-bin\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-system-cni-dir\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-log-socket\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647352 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-etc-kubernetes\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647363 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-systemd-units\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647352 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-os-release\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647395 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-cni-bin\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647402 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-log-socket\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647405 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-system-cni-dir\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-run-ovn-kubernetes\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.653038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-cni-netd\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovn-node-metrics-cert\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647505 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-run-ovn-kubernetes\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647519 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-cni-netd\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-kubelet\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-host\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647645 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-kubelet\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647677 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-host\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfq6h\" (UniqueName: \"kubernetes.io/projected/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-kube-api-access-bfq6h\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/478986e2-196e-4b09-8536-4f2d903d6c39-konnectivity-ca\") pod \"konnectivity-agent-8qrzv\" (UID: \"478986e2-196e-4b09-8536-4f2d903d6c39\") " pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-systemd\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacbedcb-b268-4879-a37b-8dc034a11e65-tmp\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lktb\" (UniqueName: \"kubernetes.io/projected/cacbedcb-b268-4879-a37b-8dc034a11e65-kube-api-access-5lktb\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-cni-bin\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysconfig\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.654784 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647851 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-systemd\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-kubernetes\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647898 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647999 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-kubernetes\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.647999 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysconfig\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648035 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-cni-bin\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovnkube-script-lib\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-device-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-etc-selinux\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648303 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/478986e2-196e-4b09-8536-4f2d903d6c39-konnectivity-ca\") pod \"konnectivity-agent-8qrzv\" (UID: \"478986e2-196e-4b09-8536-4f2d903d6c39\") " pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-daemon-config\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-modprobe-d\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648374 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-device-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6ws\" (UniqueName: \"kubernetes.io/projected/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-kube-api-access-xc6ws\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.656062 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-run-netns\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2nb\" (UniqueName: \"kubernetes.io/projected/670307dd-877f-4f25-ad8c-3b6bb6ce1134-kube-api-access-5q2nb\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-kubelet-config\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj27c\" (UniqueName: \"kubernetes.io/projected/c4977763-ab27-4d4d-b91a-28e2dac29776-kube-api-access-xj27c\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-system-cni-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-var-lib-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-k8s-cni-cncf-io\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysctl-d\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovnkube-config\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4977763-ab27-4d4d-b91a-28e2dac29776-iptables-alerter-script\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc12b8c-a240-473d-a57c-b9cd825797ef-tmp-dir\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-multus-certs\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysctl-conf\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.656989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-etc-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwz6z\" (UniqueName: \"kubernetes.io/projected/ceff6ec7-0ad5-413a-b36b-124caf4231af-kube-api-access-lwz6z\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648876 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovnkube-script-lib\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648926 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-cni-multus\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-var-lib-cni-multus\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-hostroot\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648972 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-etc-selinux\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.648987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrhn\" (UniqueName: \"kubernetes.io/projected/52506102-c9ef-4fd1-b154-19c1a64604f3-kube-api-access-krrhn\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-os-release\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.649064 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.657767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-netns\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-dbus\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-cnibin\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cnibin\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-env-overrides\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4977763-ab27-4d4d-b91a-28e2dac29776-host-slash\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-ovn\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-node-log\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-systemd\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-host\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-slash\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgrz\" (UniqueName: \"kubernetes.io/projected/5b169c47-57aa-49ee-ba53-7127a0795135-kube-api-access-xpgrz\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.658918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649497 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysctl-d\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bc12b8c-a240-473d-a57c-b9cd825797ef-hosts-file\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-socket-dir-parent\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-sys\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-var-lib-kubelet\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649645 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-hostroot\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-cni-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-conf-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-run\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-lib-modules\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-tuned\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649764 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-daemon-config\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.649816 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-os-release\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650011 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-run-netns\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.650089 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:52.659707 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.650147 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs podName:670307dd-877f-4f25-ad8c-3b6bb6ce1134 nodeName:}" failed. No retries permitted until 2026-07-06 23:42:53.150128903 +0000 UTC m=+3.177034123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs") pod "network-metrics-daemon-6lqdv" (UID: "670307dd-877f-4f25-ad8c-3b6bb6ce1134") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650209 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovnkube-config\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650263 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650304 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-modprobe-d\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650357 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-kubelet-config\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650517 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-sys\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650562 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-netns\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.650619 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret podName:fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd nodeName:}" failed. No retries permitted until 2026-07-06 23:42:53.150607721 +0000 UTC m=+3.177512928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret") pod "global-pull-secret-syncer-7rs8n" (UID: "fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650614 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-system-cni-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650671 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-var-lib-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650679 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-cni-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650706 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-dbus\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:52.660516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-k8s-cni-cncf-io\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650755 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-var-lib-kubelet\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650762 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-cnibin\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650799 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bc12b8c-a240-473d-a57c-b9cd825797ef-hosts-file\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650805 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-cnibin\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650800 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-conf-dir\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650813 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4977763-ab27-4d4d-b91a-28e2dac29776-iptables-alerter-script\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650839 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-run\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650844 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650859 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-multus-socket-dir-parent\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650921 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-ovn\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650940 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-lib-modules\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.650974 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-node-log\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651102 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc12b8c-a240-473d-a57c-b9cd825797ef-tmp-dir\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651184 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52506102-c9ef-4fd1-b154-19c1a64604f3-host-run-multus-certs\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.661490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651200 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-systemd\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651298 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-sysctl-conf\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651338 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cacbedcb-b268-4879-a37b-8dc034a11e65-host\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651344 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceff6ec7-0ad5-413a-b36b-124caf4231af-env-overrides\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651370 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-slash\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651397 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4977763-ab27-4d4d-b91a-28e2dac29776-host-slash\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-sys-fs\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-serviceca\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsrv\" (UniqueName: \"kubernetes.io/projected/9bc12b8c-a240-473d-a57c-b9cd825797ef-kube-api-access-9jsrv\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52506102-c9ef-4fd1-b154-19c1a64604f3-cni-binary-copy\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/478986e2-196e-4b09-8536-4f2d903d6c39-agent-certs\") pod \"konnectivity-agent-8qrzv\" (UID: \"478986e2-196e-4b09-8536-4f2d903d6c39\") " pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651544 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-run-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-kubelet\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-socket-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.662320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651695 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651719 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-sys-fs\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651739 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceff6ec7-0ad5-413a-b36b-124caf4231af-ovn-node-metrics-cert\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651827 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-etc-openvswitch\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651876 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651913 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff6ec7-0ad5-413a-b36b-124caf4231af-host-kubelet\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.652024 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-socket-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.651621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-registration-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.652121 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-serviceca\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.652146 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b169c47-57aa-49ee-ba53-7127a0795135-registration-dir\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.652468 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52506102-c9ef-4fd1-b154-19c1a64604f3-cni-binary-copy\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.652540 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacbedcb-b268-4879-a37b-8dc034a11e65-tmp\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.655172 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cacbedcb-b268-4879-a37b-8dc034a11e65-etc-tuned\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.656191 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/478986e2-196e-4b09-8536-4f2d903d6c39-agent-certs\") pod \"konnectivity-agent-8qrzv\" (UID: \"478986e2-196e-4b09-8536-4f2d903d6c39\") " pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.659232 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:42:52.751973 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.659253 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.659265 2564 projected.go:196] Error preparing data for projected volume kube-api-access-khz9t for pod openshift-network-diagnostics/network-check-target-jj75g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:52.659316 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t podName:d4804710-463b-4d7e-b473-bc65adf9d9ae nodeName:}" failed. No retries permitted until 2026-07-06 23:42:53.159301066 +0000 UTC m=+3.186206279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-khz9t" (UniqueName: "kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t") pod "network-check-target-jj75g" (UID: "d4804710-463b-4d7e-b473-bc65adf9d9ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.659909 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfq6h\" (UniqueName: \"kubernetes.io/projected/dbafaea7-f7b6-4dd8-b32a-fd38510ed29d-kube-api-access-bfq6h\") pod \"node-ca-sd5wt\" (UID: \"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d\") " pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.660068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" event={"ID":"9064f225fd7e0619db91cfef5c8bbc9d","Type":"ContainerStarted","Data":"02ad7add0da52467a68b61807a5c0ce4ec21f66bb51222522243ffa88472727f"} Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.662729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" event={"ID":"c9d5995c2cad545e25efb8ab86dbae04","Type":"ContainerStarted","Data":"fc56ea846bb53efe8db4a2af291de9f3f89e35f711d0751bda643c73b4c30e3f"} Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.665053 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6ws\" (UniqueName: \"kubernetes.io/projected/82b7c2c3-750c-4b5f-bc86-dd23f6292ad5-kube-api-access-xc6ws\") pod \"multus-additional-cni-plugins-6rf6b\" (UID: \"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5\") " pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.665366 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsrv\" (UniqueName: \"kubernetes.io/projected/9bc12b8c-a240-473d-a57c-b9cd825797ef-kube-api-access-9jsrv\") pod \"node-resolver-brd86\" (UID: \"9bc12b8c-a240-473d-a57c-b9cd825797ef\") " pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.665978 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2nb\" (UniqueName: \"kubernetes.io/projected/670307dd-877f-4f25-ad8c-3b6bb6ce1134-kube-api-access-5q2nb\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.666368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lktb\" (UniqueName: \"kubernetes.io/projected/cacbedcb-b268-4879-a37b-8dc034a11e65-kube-api-access-5lktb\") pod \"tuned-ljs8w\" (UID: \"cacbedcb-b268-4879-a37b-8dc034a11e65\") " pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.667052 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwz6z\" (UniqueName: \"kubernetes.io/projected/ceff6ec7-0ad5-413a-b36b-124caf4231af-kube-api-access-lwz6z\") pod \"ovnkube-node-d9pgh\" (UID: \"ceff6ec7-0ad5-413a-b36b-124caf4231af\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.667725 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgrz\" (UniqueName: \"kubernetes.io/projected/5b169c47-57aa-49ee-ba53-7127a0795135-kube-api-access-xpgrz\") pod \"aws-ebs-csi-driver-node-kp9c9\" (UID: \"5b169c47-57aa-49ee-ba53-7127a0795135\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.668327 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrhn\" (UniqueName: \"kubernetes.io/projected/52506102-c9ef-4fd1-b154-19c1a64604f3-kube-api-access-krrhn\") pod \"multus-24bw4\" (UID: \"52506102-c9ef-4fd1-b154-19c1a64604f3\") " pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.753164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.683353 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj27c\" (UniqueName: \"kubernetes.io/projected/c4977763-ab27-4d4d-b91a-28e2dac29776-kube-api-access-xj27c\") pod \"iptables-alerter-4vcbf\" (UID: \"c4977763-ab27-4d4d-b91a-28e2dac29776\") " pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.868025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.768346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sd5wt" Jul 06 23:42:52.868025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.785007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" Jul 06 23:42:52.868025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.789800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" Jul 06 23:42:52.868025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.797325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:42:52.881765 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.881739 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vcbf" Jul 06 23:42:52.898387 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.898360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-brd86" Jul 06 23:42:52.904260 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.904211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-24bw4" Jul 06 23:42:52.910911 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.910889 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:42:52.917519 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:52.917500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" Jul 06 23:42:53.158178 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:53.158087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:53.158178 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:53.158132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:53.158377 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.158242 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:53.158377 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.158250 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:53.158377 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.158314 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret podName:fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd nodeName:}" failed. No retries permitted until 2026-07-06 23:42:54.158293727 +0000 UTC m=+4.185198946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret") pod "global-pull-secret-syncer-7rs8n" (UID: "fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:53.158377 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.158333 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs podName:670307dd-877f-4f25-ad8c-3b6bb6ce1134 nodeName:}" failed. No retries permitted until 2026-07-06 23:42:54.158323518 +0000 UTC m=+4.185228725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs") pod "network-metrics-daemon-6lqdv" (UID: "670307dd-877f-4f25-ad8c-3b6bb6ce1134") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:53.258860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:53.258824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:53.259038 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.259021 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:42:53.259097 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.259042 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:42:53.259097 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.259056 2564 projected.go:196] Error preparing data for projected volume kube-api-access-khz9t for pod openshift-network-diagnostics/network-check-target-jj75g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:53.259181 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.259124 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t podName:d4804710-463b-4d7e-b473-bc65adf9d9ae nodeName:}" failed. No retries permitted until 2026-07-06 23:42:54.259104547 +0000 UTC m=+4.286009767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-khz9t" (UniqueName: "kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t") pod "network-check-target-jj75g" (UID: "d4804710-463b-4d7e-b473-bc65adf9d9ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:53.489267 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:53.489167 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:37:52 +0000 UTC" deadline="2028-02-20 08:24:08.051829991 +0000 UTC" Jul 06 23:42:53.489267 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:53.489212 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14240h41m14.562620487s" Jul 06 23:42:53.648665 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:53.648621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:53.648798 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:53.648775 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:42:53.665429 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.665402 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b7c2c3_750c_4b5f_bc86_dd23f6292ad5.slice/crio-db12616103d12b10a68680f5a1a6ddea5045b358ebbebcaffcbbef2445f55835 WatchSource:0}: Error finding container db12616103d12b10a68680f5a1a6ddea5045b358ebbebcaffcbbef2445f55835: Status 404 returned error can't find the container with id db12616103d12b10a68680f5a1a6ddea5045b358ebbebcaffcbbef2445f55835 Jul 06 23:42:53.671050 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.670796 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacbedcb_b268_4879_a37b_8dc034a11e65.slice/crio-e08ee175e13c608005f8a2d5ccfaaa7513d53015016d0721363f82c65009003a WatchSource:0}: Error finding container e08ee175e13c608005f8a2d5ccfaaa7513d53015016d0721363f82c65009003a: Status 404 returned error can't find the container with id e08ee175e13c608005f8a2d5ccfaaa7513d53015016d0721363f82c65009003a Jul 06 23:42:53.671937 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.671917 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc12b8c_a240_473d_a57c_b9cd825797ef.slice/crio-420ff1fe6e9bc9a4eb6988ec35ffa5f523ea4de2ef67622451a8eff0162ea663 WatchSource:0}: Error finding container 420ff1fe6e9bc9a4eb6988ec35ffa5f523ea4de2ef67622451a8eff0162ea663: Status 404 returned error can't find the container with id 420ff1fe6e9bc9a4eb6988ec35ffa5f523ea4de2ef67622451a8eff0162ea663 Jul 06 23:42:53.673217 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.673196 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52506102_c9ef_4fd1_b154_19c1a64604f3.slice/crio-3b5c744bcad0133d2eab7746474b934d36bad90ca6c33ed473a1946f8f1cf591 WatchSource:0}: Error finding container 3b5c744bcad0133d2eab7746474b934d36bad90ca6c33ed473a1946f8f1cf591: Status 404 returned error can't find the container with id 3b5c744bcad0133d2eab7746474b934d36bad90ca6c33ed473a1946f8f1cf591 Jul 06 23:42:53.673603 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.673538 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4977763_ab27_4d4d_b91a_28e2dac29776.slice/crio-c46aec8cdc5242456a0000de606b2291e6e736a24d5422441dfaf5a1dabe895c WatchSource:0}: Error finding container c46aec8cdc5242456a0000de606b2291e6e736a24d5422441dfaf5a1dabe895c: Status 404 returned error can't find the container with id c46aec8cdc5242456a0000de606b2291e6e736a24d5422441dfaf5a1dabe895c Jul 06 23:42:53.675331 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.675123 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b169c47_57aa_49ee_ba53_7127a0795135.slice/crio-67ea4eba2fb7c21e8382bad5cffc73318ae80c31badfbb7dfb8bea28e40c8c35 WatchSource:0}: Error finding container 67ea4eba2fb7c21e8382bad5cffc73318ae80c31badfbb7dfb8bea28e40c8c35: Status 404 returned error can't find the container with id 67ea4eba2fb7c21e8382bad5cffc73318ae80c31badfbb7dfb8bea28e40c8c35 Jul 06 23:42:53.676665 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.676582 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceff6ec7_0ad5_413a_b36b_124caf4231af.slice/crio-33664b31af9e2308865f80f3468730eea4672207861f83f5a24a07ebad6ce3ec WatchSource:0}: Error finding container 33664b31af9e2308865f80f3468730eea4672207861f83f5a24a07ebad6ce3ec: Status 404 returned error can't find the container with id 33664b31af9e2308865f80f3468730eea4672207861f83f5a24a07ebad6ce3ec Jul 06 23:42:53.677679 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:42:53.677607 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbafaea7_f7b6_4dd8_b32a_fd38510ed29d.slice/crio-55639f7be85e9703b0e626aba40c9c4e5af81d8f26fc81b347e62f627bdd97e1 WatchSource:0}: Error finding container 55639f7be85e9703b0e626aba40c9c4e5af81d8f26fc81b347e62f627bdd97e1: Status 404 returned error can't find the container with id 55639f7be85e9703b0e626aba40c9c4e5af81d8f26fc81b347e62f627bdd97e1 Jul 06 23:42:53.772996 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:53.772811 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:42:54.166994 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.166915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:54.166994 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.166969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:54.167223 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.167182 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:54.167275 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.167257 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs podName:670307dd-877f-4f25-ad8c-3b6bb6ce1134 nodeName:}" failed. No retries permitted until 2026-07-06 23:42:56.16723802 +0000 UTC m=+6.194143247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs") pod "network-metrics-daemon-6lqdv" (UID: "670307dd-877f-4f25-ad8c-3b6bb6ce1134") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:54.167706 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.167686 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:54.167809 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.167746 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret podName:fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd nodeName:}" failed. No retries permitted until 2026-07-06 23:42:56.167730367 +0000 UTC m=+6.194635580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret") pod "global-pull-secret-syncer-7rs8n" (UID: "fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:54.268915 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.268288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:54.268915 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.268466 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:42:54.268915 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.268490 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:42:54.268915 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.268502 2564 projected.go:196] Error preparing data for projected volume kube-api-access-khz9t for pod openshift-network-diagnostics/network-check-target-jj75g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:54.268915 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.268559 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t podName:d4804710-463b-4d7e-b473-bc65adf9d9ae nodeName:}" failed. No retries permitted until 2026-07-06 23:42:56.268541846 +0000 UTC m=+6.295447070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-khz9t" (UniqueName: "kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t") pod "network-check-target-jj75g" (UID: "d4804710-463b-4d7e-b473-bc65adf9d9ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:54.496082 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.495894 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:37:52 +0000 UTC" deadline="2028-03-11 20:04:18.227803467 +0000 UTC" Jul 06 23:42:54.496082 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.495950 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14732h21m23.731858035s" Jul 06 23:42:54.651248 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.651203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:54.651430 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.651344 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:42:54.651814 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.651794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:54.651923 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:54.651904 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:42:54.676916 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.676879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vcbf" event={"ID":"c4977763-ab27-4d4d-b91a-28e2dac29776","Type":"ContainerStarted","Data":"c46aec8cdc5242456a0000de606b2291e6e736a24d5422441dfaf5a1dabe895c"} Jul 06 23:42:54.711363 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.711325 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" event={"ID":"5b169c47-57aa-49ee-ba53-7127a0795135","Type":"ContainerStarted","Data":"67ea4eba2fb7c21e8382bad5cffc73318ae80c31badfbb7dfb8bea28e40c8c35"} Jul 06 23:42:54.743162 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.743129 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-24bw4" event={"ID":"52506102-c9ef-4fd1-b154-19c1a64604f3","Type":"ContainerStarted","Data":"3b5c744bcad0133d2eab7746474b934d36bad90ca6c33ed473a1946f8f1cf591"} Jul 06 23:42:54.767769 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.767685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qrzv" event={"ID":"478986e2-196e-4b09-8536-4f2d903d6c39","Type":"ContainerStarted","Data":"28cf6eeae3cd0c9aada2829401b01fe5a0101c0790b70d33fb52bdc8f46222c7"} Jul 06 23:42:54.780720 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.780688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerStarted","Data":"db12616103d12b10a68680f5a1a6ddea5045b358ebbebcaffcbbef2445f55835"} Jul 06 23:42:54.801946 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.801912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" event={"ID":"9064f225fd7e0619db91cfef5c8bbc9d","Type":"ContainerStarted","Data":"fdb8139ec44cd72f9c39c1f58d321826c440457eea92fb26054722c59ca95fa1"} Jul 06 23:42:54.807734 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.807706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sd5wt" event={"ID":"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d","Type":"ContainerStarted","Data":"55639f7be85e9703b0e626aba40c9c4e5af81d8f26fc81b347e62f627bdd97e1"} Jul 06 23:42:54.818551 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.818520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"33664b31af9e2308865f80f3468730eea4672207861f83f5a24a07ebad6ce3ec"} Jul 06 23:42:54.830795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.830732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-brd86" event={"ID":"9bc12b8c-a240-473d-a57c-b9cd825797ef","Type":"ContainerStarted","Data":"420ff1fe6e9bc9a4eb6988ec35ffa5f523ea4de2ef67622451a8eff0162ea663"} Jul 06 23:42:54.850795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.850757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" event={"ID":"cacbedcb-b268-4879-a37b-8dc034a11e65","Type":"ContainerStarted","Data":"e08ee175e13c608005f8a2d5ccfaaa7513d53015016d0721363f82c65009003a"} Jul 06 23:42:54.861428 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.861398 2564 generic.go:356] "Generic (PLEG): container finished" podID="c9d5995c2cad545e25efb8ab86dbae04" containerID="c6a9d4eb4280ce6779414385d02fdef4dd817980ffac797ab48979f6afd9fa75" exitCode=0 Jul 06 23:42:54.861648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.861438 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" event={"ID":"c9d5995c2cad545e25efb8ab86dbae04","Type":"ContainerDied","Data":"c6a9d4eb4280ce6779414385d02fdef4dd817980ffac797ab48979f6afd9fa75"} Jul 06 23:42:54.877237 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:54.877180 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" podStartSLOduration=3.8771613670000002 podStartE2EDuration="3.877161367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:42:54.818447349 +0000 UTC m=+4.845352579" watchObservedRunningTime="2026-07-06 23:42:54.877161367 +0000 UTC m=+4.904066597" Jul 06 23:42:55.648623 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:55.648591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:55.649084 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:55.648734 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:42:55.887832 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:55.887656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" event={"ID":"c9d5995c2cad545e25efb8ab86dbae04","Type":"ContainerStarted","Data":"7d10b8f5330c6e9b796da46f88e39beac03ff91723f402e1ca22388cb6b4e8a9"} Jul 06 23:42:55.903687 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:55.903619 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" podStartSLOduration=4.903598246 podStartE2EDuration="4.903598246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:42:55.903325111 +0000 UTC m=+5.930230341" watchObservedRunningTime="2026-07-06 23:42:55.903598246 +0000 UTC m=+5.930503477" Jul 06 23:42:56.183986 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:56.183876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:56.183986 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:56.183928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:56.184209 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.184072 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:56.184209 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.184134 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs podName:670307dd-877f-4f25-ad8c-3b6bb6ce1134 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:00.184116217 +0000 UTC m=+10.211021438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs") pod "network-metrics-daemon-6lqdv" (UID: "670307dd-877f-4f25-ad8c-3b6bb6ce1134") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:42:56.184812 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.184425 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:56.184812 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.184489 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret podName:fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:00.184472438 +0000 UTC m=+10.211377651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret") pod "global-pull-secret-syncer-7rs8n" (UID: "fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:42:56.284352 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:56.284314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:56.284541 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.284525 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:42:56.284606 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.284551 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:42:56.284606 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.284564 2564 projected.go:196] Error preparing data for projected volume kube-api-access-khz9t for pod openshift-network-diagnostics/network-check-target-jj75g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:56.284725 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.284647 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t podName:d4804710-463b-4d7e-b473-bc65adf9d9ae nodeName:}" failed. No retries permitted until 2026-07-06 23:43:00.284614483 +0000 UTC m=+10.311519695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-khz9t" (UniqueName: "kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t") pod "network-check-target-jj75g" (UID: "d4804710-463b-4d7e-b473-bc65adf9d9ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:42:56.649068 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:56.648989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:56.649487 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.649134 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:42:56.649487 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:56.649212 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:56.649487 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:56.649322 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:42:57.649478 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:57.649234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:57.649478 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:57.649367 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:42:58.653525 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:58.653471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:42:58.653981 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:58.653589 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:42:58.653981 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:58.653471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:42:58.654101 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:58.654043 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:42:59.648533 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:42:59.648424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:42:59.648737 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:42:59.648560 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:00.219566 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:00.219528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:00.220023 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:00.219580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:00.220023 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.219693 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:43:00.220023 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.219756 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret podName:fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:08.2197383 +0000 UTC m=+18.246643524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret") pod "global-pull-secret-syncer-7rs8n" (UID: "fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:43:00.220023 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.219698 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:43:00.220023 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.219828 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs podName:670307dd-877f-4f25-ad8c-3b6bb6ce1134 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:08.219812651 +0000 UTC m=+18.246717858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs") pod "network-metrics-daemon-6lqdv" (UID: "670307dd-877f-4f25-ad8c-3b6bb6ce1134") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:43:00.320272 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:00.320177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:00.320430 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.320299 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:43:00.320430 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.320317 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:43:00.320430 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.320328 2564 projected.go:196] Error preparing data for projected volume kube-api-access-khz9t for pod openshift-network-diagnostics/network-check-target-jj75g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:43:00.320430 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.320379 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t podName:d4804710-463b-4d7e-b473-bc65adf9d9ae nodeName:}" failed. No retries permitted until 2026-07-06 23:43:08.320360072 +0000 UTC m=+18.347265283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-khz9t" (UniqueName: "kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t") pod "network-check-target-jj75g" (UID: "d4804710-463b-4d7e-b473-bc65adf9d9ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:43:00.650092 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:00.649775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:00.650092 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.649878 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:00.651049 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:00.650863 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:00.651049 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:00.651024 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:01.649130 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:01.648638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:01.649130 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:01.648757 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:02.649088 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:02.649054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:02.649263 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:02.649229 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:02.649263 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:02.649240 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:02.649704 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:02.649368 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:03.649151 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:03.649119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:03.649316 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:03.649245 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:04.649249 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:04.649206 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:04.649420 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:04.649207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:04.649420 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:04.649334 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:04.649849 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:04.649425 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:05.648435 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:05.648403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:05.648588 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:05.648520 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:06.649120 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:06.649083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:06.649522 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:06.649207 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:06.649522 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:06.649260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:06.649522 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:06.649361 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:07.648237 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:07.648199 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:07.648462 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:07.648323 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:08.279552 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:08.279513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:08.279552 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:08.279558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:08.280022 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.279685 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:43:08.280022 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.279713 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:43:08.280022 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.279751 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret podName:fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:24.279733412 +0000 UTC m=+34.306638641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret") pod "global-pull-secret-syncer-7rs8n" (UID: "fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:43:08.280022 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.279767 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs podName:670307dd-877f-4f25-ad8c-3b6bb6ce1134 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:24.279760053 +0000 UTC m=+34.306665261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs") pod "network-metrics-daemon-6lqdv" (UID: "670307dd-877f-4f25-ad8c-3b6bb6ce1134") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:43:08.380703 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:08.380663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:08.380842 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.380819 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:43:08.380842 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.380839 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:43:08.380909 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.380849 2564 projected.go:196] Error preparing data for projected volume kube-api-access-khz9t for pod openshift-network-diagnostics/network-check-target-jj75g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:43:08.380909 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.380897 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t podName:d4804710-463b-4d7e-b473-bc65adf9d9ae nodeName:}" failed. No retries permitted until 2026-07-06 23:43:24.38088471 +0000 UTC m=+34.407789916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-khz9t" (UniqueName: "kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t") pod "network-check-target-jj75g" (UID: "d4804710-463b-4d7e-b473-bc65adf9d9ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:43:08.648746 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:08.648654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:08.648893 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.648788 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:08.648893 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:08.648847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:08.648994 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:08.648957 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:09.648506 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:09.648469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:09.648920 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:09.648582 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:10.649882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:10.649842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:10.650286 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:10.649941 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:10.650286 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:10.650047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:10.650286 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:10.650162 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:11.648521 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:11.648486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:11.648718 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:11.648607 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:12.649492 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.648799 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:12.649492 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:12.649139 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:12.650185 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.650037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:12.650185 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:12.650145 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:12.921552 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.921513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sd5wt" event={"ID":"dbafaea7-f7b6-4dd8-b32a-fd38510ed29d","Type":"ContainerStarted","Data":"2007c8c93e0ae6e91d982ae789a0cd707f24636047a8ddf1dd2134a71625d1e6"} Jul 06 23:43:12.922904 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.922880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"34709afb21a9ebfda73abcecd69e7499f8cef6baae9b8742ab09f6f8c9264765"} Jul 06 23:43:12.924283 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.924249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-brd86" event={"ID":"9bc12b8c-a240-473d-a57c-b9cd825797ef","Type":"ContainerStarted","Data":"33615ac1952603348e01e73fa87010c73a9b830c8a8cf9acc59bdf09124125c4"} Jul 06 23:43:12.925860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.925840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" event={"ID":"cacbedcb-b268-4879-a37b-8dc034a11e65","Type":"ContainerStarted","Data":"6e5787ea8a23810a80a110cb7db4887737eb27d9b4431bb77a6eef96c87e7192"} Jul 06 23:43:12.927296 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.927277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" event={"ID":"5b169c47-57aa-49ee-ba53-7127a0795135","Type":"ContainerStarted","Data":"ea1df6606d250ec1f1b3ff600f5d1250c5d54a8ab8155946b27865c9559d73f5"} Jul 06 23:43:12.928683 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.928659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-24bw4" event={"ID":"52506102-c9ef-4fd1-b154-19c1a64604f3","Type":"ContainerStarted","Data":"6bf3e83883735bdb04c4adcdaebd4ce93a2e9ea59c59187e8e9ae0c26d8e1e2e"} Jul 06 23:43:12.930003 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.929980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qrzv" event={"ID":"478986e2-196e-4b09-8536-4f2d903d6c39","Type":"ContainerStarted","Data":"c90748d6668a71ca06ba57db55a29041cb2b2282e2beb9d23c1c56a2cdea0024"} Jul 06 23:43:12.933205 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.933180 2564 generic.go:356] "Generic (PLEG): container finished" podID="82b7c2c3-750c-4b5f-bc86-dd23f6292ad5" containerID="551b9b44f3cdaf1b1cdf642423729a5a7d6c475c12f4597c09e981bea72bdc7f" exitCode=0 Jul 06 23:43:12.933317 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.933216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerDied","Data":"551b9b44f3cdaf1b1cdf642423729a5a7d6c475c12f4597c09e981bea72bdc7f"} Jul 06 23:43:12.937924 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.937889 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sd5wt" podStartSLOduration=4.130014335 podStartE2EDuration="22.93787633s" totalImagesPullingTime="18.807861995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:12.936937841 +0000 UTC m=+22.963843071" watchObservedRunningTime="2026-07-06 23:43:12.93787633 +0000 UTC m=+22.964781559" Jul 06 23:43:12.953498 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.953459 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8qrzv" podStartSLOduration=12.486586639 podStartE2EDuration="22.953448432s" totalImagesPullingTime="10.466861793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:12.953442743 +0000 UTC m=+22.980347972" watchObservedRunningTime="2026-07-06 23:43:12.953448432 +0000 UTC m=+22.980353660" Jul 06 23:43:12.971337 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.971301 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-brd86" podStartSLOduration=4.157664086 podStartE2EDuration="22.9712876s" totalImagesPullingTime="18.813623514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:12.971017259 +0000 UTC m=+22.997922500" watchObservedRunningTime="2026-07-06 23:43:12.9712876 +0000 UTC m=+22.998192829" Jul 06 23:43:12.996581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:12.996536 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ljs8w" podStartSLOduration=4.181658637 podStartE2EDuration="22.996519154s" totalImagesPullingTime="18.814860517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:12.990264485 +0000 UTC m=+23.017169727" watchObservedRunningTime="2026-07-06 23:43:12.996519154 +0000 UTC m=+23.023424383" Jul 06 23:43:13.034488 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.034448 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-24bw4" podStartSLOduration=4.185965171 podStartE2EDuration="23.034433762s" totalImagesPullingTime="18.848468591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:13.033998414 +0000 UTC m=+23.060903646" watchObservedRunningTime="2026-07-06 23:43:13.034433762 +0000 UTC m=+23.061338998" Jul 06 23:43:13.648960 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.648791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:13.649147 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:13.649053 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:13.939466 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.939443 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:43:13.940151 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.939770 2564 generic.go:356] "Generic (PLEG): container finished" podID="ceff6ec7-0ad5-413a-b36b-124caf4231af" containerID="c7b8e5737972fe7648280dff383d1e9f0b237bba2b86228a3ef99d8d66217798" exitCode=1 Jul 06 23:43:13.940151 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.939872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"a4e4a490bbf84ddb1c45db86e552f8b3b386d6f59baad486ad378e0d83064c41"} Jul 06 23:43:13.940151 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.939953 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"115ef2c5f690e7b5ec5a1bcae472f41dcf8f439f7da574abb8b3e398f0516d17"} Jul 06 23:43:13.940151 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.940008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"79b30aa4125d0ca19fd32f8ddd01afe0cdf8658effe9785f88323f0b9f9d4f08"} Jul 06 23:43:13.940151 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.940062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"fb93994dd45e97405c06fc1da50d5e83e376faccb0ed450294c19b78b9f180be"} Jul 06 23:43:13.940151 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:13.940113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerDied","Data":"c7b8e5737972fe7648280dff383d1e9f0b237bba2b86228a3ef99d8d66217798"} Jul 06 23:43:14.037227 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.037202 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 23:43:14.480250 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.480076 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T23:43:14.037221731Z","UUID":"cdf31ec8-e51a-4fd5-b627-f312f9bacba7","Handler":null,"Name":"","Endpoint":""} Jul 06 23:43:14.481855 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.481831 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 23:43:14.481993 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.481875 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 23:43:14.648350 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.648263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:14.648350 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.648289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:14.648588 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:14.648394 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:14.648588 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:14.648508 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:14.782936 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.782905 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:43:14.783643 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.783600 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:43:14.943137 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.943107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vcbf" event={"ID":"c4977763-ab27-4d4d-b91a-28e2dac29776","Type":"ContainerStarted","Data":"6b2465ef47d52f7962028f5e29e05b84b56df7cfd98d9f57eebec8206e7b10ce"} Jul 06 23:43:14.944701 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.944680 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" event={"ID":"5b169c47-57aa-49ee-ba53-7127a0795135","Type":"ContainerStarted","Data":"38bd65e83c08f578a3125fe9b36b128401922f51aae42e68a35c000030161dbc"} Jul 06 23:43:14.944859 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.944849 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:43:14.945277 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.945261 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8qrzv" Jul 06 23:43:14.959050 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:14.959015 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4vcbf" podStartSLOduration=6.147500574 podStartE2EDuration="24.959003951s" totalImagesPullingTime="18.811503377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:14.958689406 +0000 UTC m=+24.985594636" watchObservedRunningTime="2026-07-06 23:43:14.959003951 +0000 UTC m=+24.985909180" Jul 06 23:43:15.648773 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:15.648509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:15.648936 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:15.648829 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:15.950518 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:15.950438 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:43:15.950980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:15.950901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"edb6884a370af12262f285d329432dd6f33d0939e0161185ef3ad863ba5d4db5"} Jul 06 23:43:15.953296 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:15.953232 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" event={"ID":"5b169c47-57aa-49ee-ba53-7127a0795135","Type":"ContainerStarted","Data":"3c71fddf5e83bf776788bfdf7b4d6a682c4db74d492b8e3d24492cc8df4c1a44"} Jul 06 23:43:15.973957 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:15.973911 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp9c9" podStartSLOduration=4.87135355 podStartE2EDuration="25.973894432s" totalImagesPullingTime="21.102540882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:15.973571798 +0000 UTC m=+26.000477052" watchObservedRunningTime="2026-07-06 23:43:15.973894432 +0000 UTC m=+26.000799660" Jul 06 23:43:16.648648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:16.648597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:16.648862 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:16.648744 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:16.648862 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:16.648800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:16.648985 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:16.648917 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:17.648562 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:17.648526 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:17.649245 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:17.648657 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:18.648978 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.648856 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:18.649423 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.648896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:18.649423 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:18.649059 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:18.649423 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:18.649163 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:18.962071 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.961989 2564 generic.go:356] "Generic (PLEG): container finished" podID="82b7c2c3-750c-4b5f-bc86-dd23f6292ad5" containerID="8a3b18527615a64268ecf5559588d97eae0bedb85b0bce3bb238b61b1ccaeaad" exitCode=0 Jul 06 23:43:18.962229 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.962075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerDied","Data":"8a3b18527615a64268ecf5559588d97eae0bedb85b0bce3bb238b61b1ccaeaad"} Jul 06 23:43:18.965613 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.965596 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:43:18.966031 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.966007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"2148abee14913613c4c30a2f5b4d89f32d4dcf064ddabde9325ad9475a7df199"} Jul 06 23:43:18.966250 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.966234 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:43:18.966325 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.966306 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:43:18.966446 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.966382 2564 scope.go:117] "RemoveContainer" containerID="c7b8e5737972fe7648280dff383d1e9f0b237bba2b86228a3ef99d8d66217798" Jul 06 23:43:18.981933 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:18.981909 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:43:19.648485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:19.648455 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:19.648799 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:19.648552 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:19.972236 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:19.972163 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:43:19.972666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:19.972482 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" event={"ID":"ceff6ec7-0ad5-413a-b36b-124caf4231af","Type":"ContainerStarted","Data":"220c2226b8a6ca3aa4f8e67b0535ded0cf1e628ba726caaad205c5e28c727f1e"} Jul 06 23:43:19.972757 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:19.972722 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:43:19.988742 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:19.988714 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:43:20.002812 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.002767 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" podStartSLOduration=11.013819083 podStartE2EDuration="30.002753321s" totalImagesPullingTime="18.988934238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:20.001016365 +0000 UTC m=+30.027921593" watchObservedRunningTime="2026-07-06 23:43:20.002753321 +0000 UTC m=+30.029658550" Jul 06 23:43:20.485406 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.485329 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7rs8n"] Jul 06 23:43:20.485563 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.485452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:20.485563 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:20.485553 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:20.488475 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.488435 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jj75g"] Jul 06 23:43:20.488583 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.488546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:20.488673 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:20.488652 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:20.489017 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.488997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6lqdv"] Jul 06 23:43:20.489104 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.489078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:20.489194 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:20.489173 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:20.976727 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.976531 2564 generic.go:356] "Generic (PLEG): container finished" podID="82b7c2c3-750c-4b5f-bc86-dd23f6292ad5" containerID="9ff3b7ed40c1f672bb42af01403b52c3346784452abb5376998aef5565aeea06" exitCode=0 Jul 06 23:43:20.976727 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:20.976616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerDied","Data":"9ff3b7ed40c1f672bb42af01403b52c3346784452abb5376998aef5565aeea06"} Jul 06 23:43:22.648548 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:22.648509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:22.648548 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:22.648509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:22.649254 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:22.648571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:22.649254 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:22.648688 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:22.649254 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:22.648720 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:22.649254 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:22.648798 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:22.983034 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:22.982949 2564 generic.go:356] "Generic (PLEG): container finished" podID="82b7c2c3-750c-4b5f-bc86-dd23f6292ad5" containerID="264d7634f89abdccd1544a5a0f02bc9781d5580d5860b09147a79e7f6543ce39" exitCode=0 Jul 06 23:43:22.983034 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:22.982990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerDied","Data":"264d7634f89abdccd1544a5a0f02bc9781d5580d5860b09147a79e7f6543ce39"} Jul 06 23:43:24.295813 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:24.295782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:24.296269 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:24.295831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:24.296269 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.295904 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:43:24.296269 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.295963 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:43:24.296269 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.295974 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret podName:fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:56.295955743 +0000 UTC m=+66.322860963 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret") pod "global-pull-secret-syncer-7rs8n" (UID: "fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:43:24.296269 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.296009 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs podName:670307dd-877f-4f25-ad8c-3b6bb6ce1134 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:56.29599292 +0000 UTC m=+66.322898132 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs") pod "network-metrics-daemon-6lqdv" (UID: "670307dd-877f-4f25-ad8c-3b6bb6ce1134") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:43:24.397068 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:24.397031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:24.397246 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.397180 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:43:24.397246 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.397200 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:43:24.397246 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.397214 2564 projected.go:196] Error preparing data for projected volume kube-api-access-khz9t for pod openshift-network-diagnostics/network-check-target-jj75g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:43:24.397414 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.397274 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t podName:d4804710-463b-4d7e-b473-bc65adf9d9ae nodeName:}" failed. No retries permitted until 2026-07-06 23:43:56.397259198 +0000 UTC m=+66.424164409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-khz9t" (UniqueName: "kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t") pod "network-check-target-jj75g" (UID: "d4804710-463b-4d7e-b473-bc65adf9d9ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:43:24.649112 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:24.649020 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:24.649279 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:24.649033 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:24.649279 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:24.649155 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:24.649279 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.649152 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj75g" podUID="d4804710-463b-4d7e-b473-bc65adf9d9ae" Jul 06 23:43:24.649279 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.649249 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lqdv" podUID="670307dd-877f-4f25-ad8c-3b6bb6ce1134" Jul 06 23:43:24.652701 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:24.650222 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rs8n" podUID="fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd" Jul 06 23:43:25.307932 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.307900 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeReady" Jul 06 23:43:25.308390 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.308051 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 23:43:25.346213 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.346176 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76ccc5f9cc-m6kjg"] Jul 06 23:43:25.349779 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.349760 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.352260 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.352240 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 23:43:25.352760 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.352743 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 23:43:25.352876 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.352861 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lx775\"" Jul 06 23:43:25.352994 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.352979 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 23:43:25.358874 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.358856 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wpbdz"] Jul 06 23:43:25.361708 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.361692 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 23:43:25.362405 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.362390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.364318 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.364167 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b49kq"] Jul 06 23:43:25.366101 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.366072 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6mrd\"" Jul 06 23:43:25.366817 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.366798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 23:43:25.368174 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.367571 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76ccc5f9cc-m6kjg"] Jul 06 23:43:25.368174 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.367693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.370012 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.369842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 23:43:25.370247 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.370226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 23:43:25.370853 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.370731 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7np4z\"" Jul 06 23:43:25.373227 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.373198 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 23:43:25.373375 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.373352 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 23:43:25.377874 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.377853 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wpbdz"] Jul 06 23:43:25.399271 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.399237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b49kq"] Jul 06 23:43:25.404500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-ca-trust-extracted\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.404654 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-trusted-ca\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.404654 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-installation-pull-secrets\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.404785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5287318-0b55-4737-ac2f-be4aa83b35ba-cert\") pod \"ingress-canary-b49kq\" (UID: \"d5287318-0b55-4737-ac2f-be4aa83b35ba\") " pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.404785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58n4h\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-kube-api-access-58n4h\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.404785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpq5\" (UniqueName: \"kubernetes.io/projected/7d410004-c4aa-4f89-9376-6ca83adf6ace-kube-api-access-hbpq5\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.404785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj68q\" (UniqueName: \"kubernetes.io/projected/d5287318-0b55-4737-ac2f-be4aa83b35ba-kube-api-access-bj68q\") pod \"ingress-canary-b49kq\" (UID: \"d5287318-0b55-4737-ac2f-be4aa83b35ba\") " pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.404971 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.404971 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-image-registry-private-configuration\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.404971 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.404971 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-certificates\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.405164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d410004-c4aa-4f89-9376-6ca83adf6ace-tmp-dir\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.405164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.404999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-bound-sa-token\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.405164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.405022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d410004-c4aa-4f89-9376-6ca83adf6ace-config-volume\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.505972 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.505937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d410004-c4aa-4f89-9376-6ca83adf6ace-tmp-dir\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.505972 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.505971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-bound-sa-token\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506177 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.505991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d410004-c4aa-4f89-9376-6ca83adf6ace-config-volume\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.506177 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-ca-trust-extracted\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506177 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-trusted-ca\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506295 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-installation-pull-secrets\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506295 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5287318-0b55-4737-ac2f-be4aa83b35ba-cert\") pod \"ingress-canary-b49kq\" (UID: \"d5287318-0b55-4737-ac2f-be4aa83b35ba\") " pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.506384 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58n4h\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-kube-api-access-58n4h\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506384 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506336 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d410004-c4aa-4f89-9376-6ca83adf6ace-tmp-dir\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.506502 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506373 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-ca-trust-extracted\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506552 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpq5\" (UniqueName: \"kubernetes.io/projected/7d410004-c4aa-4f89-9376-6ca83adf6ace-kube-api-access-hbpq5\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.506552 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj68q\" (UniqueName: \"kubernetes.io/projected/d5287318-0b55-4737-ac2f-be4aa83b35ba-kube-api-access-bj68q\") pod \"ingress-canary-b49kq\" (UID: \"d5287318-0b55-4737-ac2f-be4aa83b35ba\") " pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.506670 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.506670 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-image-registry-private-configuration\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506670 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506826 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-certificates\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.506826 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.506692 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d410004-c4aa-4f89-9376-6ca83adf6ace-config-volume\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.506826 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:25.506749 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:43:25.506826 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:25.506809 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:43:25.506826 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:25.506822 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ccc5f9cc-m6kjg: secret "image-registry-tls" not found Jul 06 23:43:25.507057 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:25.506839 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls podName:7d410004-c4aa-4f89-9376-6ca83adf6ace nodeName:}" failed. No retries permitted until 2026-07-06 23:43:26.006803401 +0000 UTC m=+36.033708614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls") pod "dns-default-wpbdz" (UID: "7d410004-c4aa-4f89-9376-6ca83adf6ace") : secret "dns-default-metrics-tls" not found Jul 06 23:43:25.507057 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:25.506925 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls podName:1e1fa4c9-5f42-4ee6-89ab-7f735b43977e nodeName:}" failed. No retries permitted until 2026-07-06 23:43:26.006906331 +0000 UTC m=+36.033811540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls") pod "image-registry-76ccc5f9cc-m6kjg" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e") : secret "image-registry-tls" not found Jul 06 23:43:25.507250 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.507227 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-certificates\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.507315 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.507291 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-trusted-ca\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.510332 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.510222 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-installation-pull-secrets\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.510402 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.510232 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-image-registry-private-configuration\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.510402 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.510319 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5287318-0b55-4737-ac2f-be4aa83b35ba-cert\") pod \"ingress-canary-b49kq\" (UID: \"d5287318-0b55-4737-ac2f-be4aa83b35ba\") " pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.535581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.535552 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58n4h\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-kube-api-access-58n4h\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.535713 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.535556 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-bound-sa-token\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:25.535899 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.535877 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpq5\" (UniqueName: \"kubernetes.io/projected/7d410004-c4aa-4f89-9376-6ca83adf6ace-kube-api-access-hbpq5\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:25.536019 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.536004 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj68q\" (UniqueName: \"kubernetes.io/projected/d5287318-0b55-4737-ac2f-be4aa83b35ba-kube-api-access-bj68q\") pod \"ingress-canary-b49kq\" (UID: \"d5287318-0b55-4737-ac2f-be4aa83b35ba\") " pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.631129 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.631006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:25.680748 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.680725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b49kq" Jul 06 23:43:25.735615 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.735524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:25.830130 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.830096 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b49kq"] Jul 06 23:43:25.835434 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:25.835411 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5287318_0b55_4737_ac2f_be4aa83b35ba.slice/crio-3a190c87dd543756aca342481d4173ffb568ad5a22bd52238a8f14c8579bc69b WatchSource:0}: Error finding container 3a190c87dd543756aca342481d4173ffb568ad5a22bd52238a8f14c8579bc69b: Status 404 returned error can't find the container with id 3a190c87dd543756aca342481d4173ffb568ad5a22bd52238a8f14c8579bc69b Jul 06 23:43:25.992034 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:25.991951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b49kq" event={"ID":"d5287318-0b55-4737-ac2f-be4aa83b35ba","Type":"ContainerStarted","Data":"3a190c87dd543756aca342481d4173ffb568ad5a22bd52238a8f14c8579bc69b"} Jul 06 23:43:26.011409 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.011369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:26.011577 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.011426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:26.011577 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:26.011539 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:43:26.011736 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:26.011603 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:43:26.011736 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:26.011621 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ccc5f9cc-m6kjg: secret "image-registry-tls" not found Jul 06 23:43:26.011736 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:26.011613 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls podName:7d410004-c4aa-4f89-9376-6ca83adf6ace nodeName:}" failed. No retries permitted until 2026-07-06 23:43:27.011593346 +0000 UTC m=+37.038498575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls") pod "dns-default-wpbdz" (UID: "7d410004-c4aa-4f89-9376-6ca83adf6ace") : secret "dns-default-metrics-tls" not found Jul 06 23:43:26.011736 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:26.011717 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls podName:1e1fa4c9-5f42-4ee6-89ab-7f735b43977e nodeName:}" failed. No retries permitted until 2026-07-06 23:43:27.011696084 +0000 UTC m=+37.038601305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls") pod "image-registry-76ccc5f9cc-m6kjg" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e") : secret "image-registry-tls" not found Jul 06 23:43:26.024952 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.024914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:26.124819 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.124771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:26.649205 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.649169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:26.649686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.649169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:26.649686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.649188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:26.652820 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.652496 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:43:26.652820 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.652642 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:43:26.652820 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.652654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2m52g\"" Jul 06 23:43:26.652820 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.652749 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xv6vb\"" Jul 06 23:43:26.653082 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.652916 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:43:26.653082 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:26.652953 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:43:27.019693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.019600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:27.019693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.019684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:27.019899 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:27.019774 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:43:27.019899 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:27.019821 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:43:27.019899 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:27.019838 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ccc5f9cc-m6kjg: secret "image-registry-tls" not found Jul 06 23:43:27.019899 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:27.019841 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls podName:7d410004-c4aa-4f89-9376-6ca83adf6ace nodeName:}" failed. No retries permitted until 2026-07-06 23:43:29.019821189 +0000 UTC m=+39.046726441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls") pod "dns-default-wpbdz" (UID: "7d410004-c4aa-4f89-9376-6ca83adf6ace") : secret "dns-default-metrics-tls" not found Jul 06 23:43:27.019899 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:27.019901 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls podName:1e1fa4c9-5f42-4ee6-89ab-7f735b43977e nodeName:}" failed. No retries permitted until 2026-07-06 23:43:29.019885109 +0000 UTC m=+39.046790318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls") pod "image-registry-76ccc5f9cc-m6kjg" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e") : secret "image-registry-tls" not found Jul 06 23:43:27.908585 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.908555 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jzx9r"] Jul 06 23:43:27.911455 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.911429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:27.913638 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.913606 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 23:43:27.914652 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.914422 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 23:43:27.914652 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.914540 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c4tcd\"" Jul 06 23:43:27.914652 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.914563 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 23:43:27.914846 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.914624 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 23:43:27.918799 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.918780 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 23:43:27.921284 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:27.921261 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jzx9r"] Jul 06 23:43:28.000398 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.000155 2564 generic.go:356] "Generic (PLEG): container finished" podID="82b7c2c3-750c-4b5f-bc86-dd23f6292ad5" containerID="dd340aba65b1d774f5bc5db8d8c4c4f8be37fa817babf229c664ff46e2e06300" exitCode=0 Jul 06 23:43:28.000398 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.000244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerDied","Data":"dd340aba65b1d774f5bc5db8d8c4c4f8be37fa817babf229c664ff46e2e06300"} Jul 06 23:43:28.009656 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.008568 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf"] Jul 06 23:43:28.012739 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.012716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" Jul 06 23:43:28.024318 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.024295 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc"] Jul 06 23:43:28.026278 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.026251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:43:28.026377 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.026260 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-brzxq\"" Jul 06 23:43:28.026469 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.026447 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 23:43:28.028301 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.028280 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d86c67597-d9q6b"] Jul 06 23:43:28.028574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.028555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.029230 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.029201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/438b86b7-b5b2-4243-a133-af4c044f9b57-tmp\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.029344 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.029326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438b86b7-b5b2-4243-a133-af4c044f9b57-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.029411 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.029356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438b86b7-b5b2-4243-a133-af4c044f9b57-serving-cert\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.029411 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.029389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/438b86b7-b5b2-4243-a133-af4c044f9b57-snapshots\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.029516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.029415 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438b86b7-b5b2-4243-a133-af4c044f9b57-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.029516 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.029456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkd5r\" (UniqueName: \"kubernetes.io/projected/438b86b7-b5b2-4243-a133-af4c044f9b57-kube-api-access-jkd5r\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.031830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.030965 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 23:43:28.031830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.031386 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 23:43:28.031830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.031482 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 23:43:28.032450 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.032341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8hw9l\"" Jul 06 23:43:28.033323 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.032672 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sf98j"] Jul 06 23:43:28.033323 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.032800 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 23:43:28.033478 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.033448 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.035834 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.035814 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 23:43:28.035947 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.035909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf"] Jul 06 23:43:28.036018 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.036005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.036072 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.036055 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 23:43:28.037008 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.036586 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 23:43:28.037008 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.036791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 23:43:28.037008 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.036816 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 23:43:28.037008 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.036874 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8z6zv\"" Jul 06 23:43:28.037008 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.036980 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 23:43:28.037928 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.037903 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 23:43:28.038468 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.038348 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:43:28.038576 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.038531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sf98j"] Jul 06 23:43:28.039667 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.039017 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ljrhb\"" Jul 06 23:43:28.039667 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.039033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 23:43:28.039667 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.039545 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc"] Jul 06 23:43:28.039860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.039754 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 23:43:28.046853 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.046833 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 23:43:28.060515 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.060465 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d86c67597-d9q6b"] Jul 06 23:43:28.113509 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.113482 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n"] Jul 06 23:43:28.117168 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.117153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:28.119212 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.119190 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 23:43:28.119399 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.119376 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:43:28.119533 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.119471 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vg7xt\"" Jul 06 23:43:28.119721 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.119707 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 23:43:28.129314 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.129293 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n"] Jul 06 23:43:28.130150 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-trusted-ca\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.130263 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl77n\" (UniqueName: \"kubernetes.io/projected/870abde4-501e-4a82-837d-abbb43a831cd-kube-api-access-sl77n\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.130263 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.130263 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sh6g\" (UniqueName: \"kubernetes.io/projected/89b36ac3-0f96-40df-9c23-718e2d608e87-kube-api-access-5sh6g\") pod \"volume-data-source-validator-5d456fdbcd-ghzlf\" (UID: \"89b36ac3-0f96-40df-9c23-718e2d608e87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" Jul 06 23:43:28.130263 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-stats-auth\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.130263 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438b86b7-b5b2-4243-a133-af4c044f9b57-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnd5\" (UniqueName: \"kubernetes.io/projected/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-kube-api-access-jfnd5\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-config\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6jh\" (UniqueName: \"kubernetes.io/projected/0522ae39-76b7-493c-9a01-2913bf880b08-kube-api-access-xw6jh\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/870abde4-501e-4a82-837d-abbb43a831cd-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/438b86b7-b5b2-4243-a133-af4c044f9b57-snapshots\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130401 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-serving-cert\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438b86b7-b5b2-4243-a133-af4c044f9b57-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438b86b7-b5b2-4243-a133-af4c044f9b57-serving-cert\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-default-certificate\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkd5r\" (UniqueName: \"kubernetes.io/projected/438b86b7-b5b2-4243-a133-af4c044f9b57-kube-api-access-jkd5r\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.130686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/438b86b7-b5b2-4243-a133-af4c044f9b57-tmp\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.131357 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.130981 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/438b86b7-b5b2-4243-a133-af4c044f9b57-tmp\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.131908 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.131417 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/438b86b7-b5b2-4243-a133-af4c044f9b57-snapshots\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.131992 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.131921 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438b86b7-b5b2-4243-a133-af4c044f9b57-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.132451 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.132409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/438b86b7-b5b2-4243-a133-af4c044f9b57-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.136050 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.136022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/438b86b7-b5b2-4243-a133-af4c044f9b57-serving-cert\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.144836 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.144805 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkd5r\" (UniqueName: \"kubernetes.io/projected/438b86b7-b5b2-4243-a133-af4c044f9b57-kube-api-access-jkd5r\") pod \"insights-operator-5c847dcfc4-jzx9r\" (UID: \"438b86b7-b5b2-4243-a133-af4c044f9b57\") " pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.207842 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.207809 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:28.231153 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-config\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.231320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6jh\" (UniqueName: \"kubernetes.io/projected/0522ae39-76b7-493c-9a01-2913bf880b08-kube-api-access-xw6jh\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.231320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/870abde4-501e-4a82-837d-abbb43a831cd-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.231320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.231320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-serving-cert\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.231539 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:28.231539 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231368 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-default-certificate\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.231539 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.231539 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6cks\" (UniqueName: \"kubernetes.io/projected/dc6a3492-579a-4059-8ca1-7023d0b1a28a-kube-api-access-n6cks\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:28.231539 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-trusted-ca\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.231539 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl77n\" (UniqueName: \"kubernetes.io/projected/870abde4-501e-4a82-837d-abbb43a831cd-kube-api-access-sl77n\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.231852 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.231548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.231852 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.231740 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:43:28.231852 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.231778 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:28.731752824 +0000 UTC m=+38.758658053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : configmap references non-existent config key: service-ca.crt Jul 06 23:43:28.232005 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.231888 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:28.731871522 +0000 UTC m=+38.758776743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : secret "router-metrics-certs-default" not found Jul 06 23:43:28.232086 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.232057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sh6g\" (UniqueName: \"kubernetes.io/projected/89b36ac3-0f96-40df-9c23-718e2d608e87-kube-api-access-5sh6g\") pod \"volume-data-source-validator-5d456fdbcd-ghzlf\" (UID: \"89b36ac3-0f96-40df-9c23-718e2d608e87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" Jul 06 23:43:28.232148 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.232113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-config\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.232148 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.232120 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-stats-auth\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.232249 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.232169 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:28.232249 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.232187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnd5\" (UniqueName: \"kubernetes.io/projected/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-kube-api-access-jfnd5\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.232249 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.232232 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls podName:870abde4-501e-4a82-837d-abbb43a831cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:28.732218118 +0000 UTC m=+38.759123324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-84wcc" (UID: "870abde4-501e-4a82-837d-abbb43a831cd") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:28.233262 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.233221 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-trusted-ca\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.234124 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.234078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/870abde4-501e-4a82-837d-abbb43a831cd-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.235159 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.235114 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-stats-auth\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.235886 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.235846 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-default-certificate\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.237761 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.237485 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-serving-cert\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.242109 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.242078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6jh\" (UniqueName: \"kubernetes.io/projected/0522ae39-76b7-493c-9a01-2913bf880b08-kube-api-access-xw6jh\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.242940 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.242917 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnd5\" (UniqueName: \"kubernetes.io/projected/1e5f900f-d24d-4c1f-a353-b1f26165a5eb-kube-api-access-jfnd5\") pod \"console-operator-68b757fb67-sf98j\" (UID: \"1e5f900f-d24d-4c1f-a353-b1f26165a5eb\") " pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.244162 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.244141 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sh6g\" (UniqueName: \"kubernetes.io/projected/89b36ac3-0f96-40df-9c23-718e2d608e87-kube-api-access-5sh6g\") pod \"volume-data-source-validator-5d456fdbcd-ghzlf\" (UID: \"89b36ac3-0f96-40df-9c23-718e2d608e87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" Jul 06 23:43:28.245800 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.245778 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl77n\" (UniqueName: \"kubernetes.io/projected/870abde4-501e-4a82-837d-abbb43a831cd-kube-api-access-sl77n\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.246406 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.246385 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" Jul 06 23:43:28.300958 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.300912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:28.324405 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.324374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" Jul 06 23:43:28.333451 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.333389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:28.333582 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.333483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6cks\" (UniqueName: \"kubernetes.io/projected/dc6a3492-579a-4059-8ca1-7023d0b1a28a-kube-api-access-n6cks\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:28.333664 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.333638 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:43:28.333746 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.333717 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls podName:dc6a3492-579a-4059-8ca1-7023d0b1a28a nodeName:}" failed. No retries permitted until 2026-07-06 23:43:28.833695332 +0000 UTC m=+38.860600541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-f227n" (UID: "dc6a3492-579a-4059-8ca1-7023d0b1a28a") : secret "samples-operator-tls" not found Jul 06 23:43:28.349226 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.349170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6cks\" (UniqueName: \"kubernetes.io/projected/dc6a3492-579a-4059-8ca1-7023d0b1a28a-kube-api-access-n6cks\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:28.364594 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.363036 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:28.431081 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.430859 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jzx9r"] Jul 06 23:43:28.503489 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.494862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf"] Jul 06 23:43:28.508624 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:28.508595 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b36ac3_0f96_40df_9c23_718e2d608e87.slice/crio-5cfafa3a63933a256ae43209704a2cc51b78f026a16ebddb36946ae203b50da1 WatchSource:0}: Error finding container 5cfafa3a63933a256ae43209704a2cc51b78f026a16ebddb36946ae203b50da1: Status 404 returned error can't find the container with id 5cfafa3a63933a256ae43209704a2cc51b78f026a16ebddb36946ae203b50da1 Jul 06 23:43:28.542644 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.542605 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sf98j"] Jul 06 23:43:28.545969 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:28.545938 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5f900f_d24d_4c1f_a353_b1f26165a5eb.slice/crio-2f49692c3ba5e5e668fcdec520013b1c2d19b45d4ee14cea4c81b863232ed948 WatchSource:0}: Error finding container 2f49692c3ba5e5e668fcdec520013b1c2d19b45d4ee14cea4c81b863232ed948: Status 404 returned error can't find the container with id 2f49692c3ba5e5e668fcdec520013b1c2d19b45d4ee14cea4c81b863232ed948 Jul 06 23:43:28.738827 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.738774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:28.738991 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.738854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.738991 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.738936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:28.738991 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.738951 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:28.739149 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.739017 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls podName:870abde4-501e-4a82-837d-abbb43a831cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:29.738999583 +0000 UTC m=+39.765904790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-84wcc" (UID: "870abde4-501e-4a82-837d-abbb43a831cd") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:28.739149 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.739059 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:29.739048118 +0000 UTC m=+39.765953325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : configmap references non-existent config key: service-ca.crt Jul 06 23:43:28.739149 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.739111 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:43:28.739149 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.739132 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:29.739125362 +0000 UTC m=+39.766030569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : secret "router-metrics-certs-default" not found Jul 06 23:43:28.840170 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:28.840129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:28.840328 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.840289 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:43:28.840384 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:28.840364 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls podName:dc6a3492-579a-4059-8ca1-7023d0b1a28a nodeName:}" failed. No retries permitted until 2026-07-06 23:43:29.840343362 +0000 UTC m=+39.867248589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-f227n" (UID: "dc6a3492-579a-4059-8ca1-7023d0b1a28a") : secret "samples-operator-tls" not found Jul 06 23:43:29.005342 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.005208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" event={"ID":"1e5f900f-d24d-4c1f-a353-b1f26165a5eb","Type":"ContainerStarted","Data":"2f49692c3ba5e5e668fcdec520013b1c2d19b45d4ee14cea4c81b863232ed948"} Jul 06 23:43:29.011955 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.011117 2564 generic.go:356] "Generic (PLEG): container finished" podID="82b7c2c3-750c-4b5f-bc86-dd23f6292ad5" containerID="5f9a49c0960004ac342a0c9686aac836c860466f686ce7a6919b4eb129e2d1aa" exitCode=0 Jul 06 23:43:29.011955 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.011241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerDied","Data":"5f9a49c0960004ac342a0c9686aac836c860466f686ce7a6919b4eb129e2d1aa"} Jul 06 23:43:29.015922 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.015892 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" event={"ID":"89b36ac3-0f96-40df-9c23-718e2d608e87","Type":"ContainerStarted","Data":"5cfafa3a63933a256ae43209704a2cc51b78f026a16ebddb36946ae203b50da1"} Jul 06 23:43:29.018079 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.018016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" event={"ID":"438b86b7-b5b2-4243-a133-af4c044f9b57","Type":"ContainerStarted","Data":"598afa0d02f6b909cf1df6077c1f0865e4709651735a2c5173ddfb83591720e1"} Jul 06 23:43:29.042598 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.042373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:29.042773 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.042755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:29.043406 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.043386 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:43:29.043489 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.043448 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls podName:7d410004-c4aa-4f89-9376-6ca83adf6ace nodeName:}" failed. No retries permitted until 2026-07-06 23:43:33.043429995 +0000 UTC m=+43.070335202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls") pod "dns-default-wpbdz" (UID: "7d410004-c4aa-4f89-9376-6ca83adf6ace") : secret "dns-default-metrics-tls" not found Jul 06 23:43:29.044515 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.044419 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:43:29.044515 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.044439 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ccc5f9cc-m6kjg: secret "image-registry-tls" not found Jul 06 23:43:29.044515 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.044483 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls podName:1e1fa4c9-5f42-4ee6-89ab-7f735b43977e nodeName:}" failed. No retries permitted until 2026-07-06 23:43:33.044467045 +0000 UTC m=+43.071372269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls") pod "image-registry-76ccc5f9cc-m6kjg" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e") : secret "image-registry-tls" not found Jul 06 23:43:29.423401 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.423353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:29.537455 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.537183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:29.635149 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.635116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:29.735488 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.735406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:29.749473 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.749425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:29.749752 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.749515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:29.749752 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.749563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:29.749752 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.749726 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:43:29.749977 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.749784 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:31.749766572 +0000 UTC m=+41.776671784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : secret "router-metrics-certs-default" not found Jul 06 23:43:29.750248 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.750161 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:29.750248 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.750175 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:31.750155833 +0000 UTC m=+41.777061057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : configmap references non-existent config key: service-ca.crt Jul 06 23:43:29.750248 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.750222 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls podName:870abde4-501e-4a82-837d-abbb43a831cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:31.750202196 +0000 UTC m=+41.777107422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-84wcc" (UID: "870abde4-501e-4a82-837d-abbb43a831cd") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:29.836377 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.836346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:29.850537 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:29.850503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:29.850728 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.850659 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:43:29.850797 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:29.850736 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls podName:dc6a3492-579a-4059-8ca1-7023d0b1a28a nodeName:}" failed. No retries permitted until 2026-07-06 23:43:31.850703089 +0000 UTC m=+41.877608299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-f227n" (UID: "dc6a3492-579a-4059-8ca1-7023d0b1a28a") : secret "samples-operator-tls" not found Jul 06 23:43:30.025468 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.025434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" event={"ID":"82b7c2c3-750c-4b5f-bc86-dd23f6292ad5","Type":"ContainerStarted","Data":"3c3919202ddcc35fe659a6fbbdf90711a326155617f56d482a0994181a70ec4c"} Jul 06 23:43:30.049322 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.049272 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6rf6b" podStartSLOduration=8.588760615 podStartE2EDuration="40.04925564s" totalImagesPullingTime="31.460495025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:30.048530344 +0000 UTC m=+40.075435589" watchObservedRunningTime="2026-07-06 23:43:30.04925564 +0000 UTC m=+40.076160869" Jul 06 23:43:30.184478 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.184442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:30.284201 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.284121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:30.385025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.384994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:30.442220 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.442177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:30.776136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.776099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:30.829826 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:30.829791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:31.768327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.768288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:31.768700 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.768344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:31.768700 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.768405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:31.768700 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:31.768447 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:31.768700 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:31.768489 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:43:31.768700 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:31.768505 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls podName:870abde4-501e-4a82-837d-abbb43a831cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:35.768490333 +0000 UTC m=+45.795395540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-84wcc" (UID: "870abde4-501e-4a82-837d-abbb43a831cd") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:31.768700 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:31.768520 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:35.768512895 +0000 UTC m=+45.795418101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : configmap references non-existent config key: service-ca.crt Jul 06 23:43:31.768700 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:31.768539 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:35.76852679 +0000 UTC m=+45.795432002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : secret "router-metrics-certs-default" not found Jul 06 23:43:31.868969 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.868937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:31.869183 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:31.869089 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:43:31.869183 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:31.869154 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls podName:dc6a3492-579a-4059-8ca1-7023d0b1a28a nodeName:}" failed. No retries permitted until 2026-07-06 23:43:35.869138757 +0000 UTC m=+45.896043964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-f227n" (UID: "dc6a3492-579a-4059-8ca1-7023d0b1a28a") : secret "samples-operator-tls" not found Jul 06 23:43:31.941328 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.941288 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk"] Jul 06 23:43:31.956126 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.956102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk"] Jul 06 23:43:31.956267 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.956195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:31.958997 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.958978 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:43:31.958997 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.958990 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 23:43:31.959815 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.959792 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sqbj9\"" Jul 06 23:43:31.959906 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.959890 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 23:43:31.960001 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:31.959985 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 23:43:32.070778 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.070745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/539120a9-c4eb-424e-942b-f7f2047712b7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.070894 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.070877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcvf\" (UniqueName: \"kubernetes.io/projected/539120a9-c4eb-424e-942b-f7f2047712b7-kube-api-access-rqcvf\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.070931 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.070908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/539120a9-c4eb-424e-942b-f7f2047712b7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.172267 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.172236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcvf\" (UniqueName: \"kubernetes.io/projected/539120a9-c4eb-424e-942b-f7f2047712b7-kube-api-access-rqcvf\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.172267 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.172274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/539120a9-c4eb-424e-942b-f7f2047712b7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.172457 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.172365 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/539120a9-c4eb-424e-942b-f7f2047712b7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.172960 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.172942 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/539120a9-c4eb-424e-942b-f7f2047712b7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.175308 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.175290 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/539120a9-c4eb-424e-942b-f7f2047712b7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.184840 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.184809 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcvf\" (UniqueName: \"kubernetes.io/projected/539120a9-c4eb-424e-942b-f7f2047712b7-kube-api-access-rqcvf\") pod \"kube-storage-version-migrator-operator-846bf668f7-b97bk\" (UID: \"539120a9-c4eb-424e-942b-f7f2047712b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.265492 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.265469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" Jul 06 23:43:32.431192 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:32.431157 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk"] Jul 06 23:43:32.434087 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:32.434064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539120a9_c4eb_424e_942b_f7f2047712b7.slice/crio-f6b47855b602bd129422fbb5cc16d060016e24e2796fe653344251a7dee6d193 WatchSource:0}: Error finding container f6b47855b602bd129422fbb5cc16d060016e24e2796fe653344251a7dee6d193: Status 404 returned error can't find the container with id f6b47855b602bd129422fbb5cc16d060016e24e2796fe653344251a7dee6d193 Jul 06 23:43:33.038035 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.037977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" event={"ID":"89b36ac3-0f96-40df-9c23-718e2d608e87","Type":"ContainerStarted","Data":"0770cd37f92dafb749e78aef53d5410dec14222684f4ad76d7dfd3cda6efdc51"} Jul 06 23:43:33.039838 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.039799 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" event={"ID":"438b86b7-b5b2-4243-a133-af4c044f9b57","Type":"ContainerStarted","Data":"a278a79e8c69016cd379e78fbec33c58655e95c899aca224a3962aa24f2cb9a1"} Jul 06 23:43:33.041500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.041477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b49kq" event={"ID":"d5287318-0b55-4737-ac2f-be4aa83b35ba","Type":"ContainerStarted","Data":"392cedd41e57a4cfffd2e662a1c5626284b576bf02672ae6935409973eb4d244"} Jul 06 23:43:33.042775 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.042749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" event={"ID":"539120a9-c4eb-424e-942b-f7f2047712b7","Type":"ContainerStarted","Data":"f6b47855b602bd129422fbb5cc16d060016e24e2796fe653344251a7dee6d193"} Jul 06 23:43:33.044421 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.044401 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/0.log" Jul 06 23:43:33.044521 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.044443 2564 generic.go:356] "Generic (PLEG): container finished" podID="1e5f900f-d24d-4c1f-a353-b1f26165a5eb" containerID="33d0be19193c2fa25ce7a699e1f9229bb7754ed1c2205243439c08d2904ec1a1" exitCode=255 Jul 06 23:43:33.044521 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.044469 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" event={"ID":"1e5f900f-d24d-4c1f-a353-b1f26165a5eb","Type":"ContainerDied","Data":"33d0be19193c2fa25ce7a699e1f9229bb7754ed1c2205243439c08d2904ec1a1"} Jul 06 23:43:33.044724 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.044707 2564 scope.go:117] "RemoveContainer" containerID="33d0be19193c2fa25ce7a699e1f9229bb7754ed1c2205243439c08d2904ec1a1" Jul 06 23:43:33.053826 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.053767 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ghzlf" podStartSLOduration=2.301430051 podStartE2EDuration="6.053753801s" totalImagesPullingTime="3.75232375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:33.052878262 +0000 UTC m=+43.079783504" watchObservedRunningTime="2026-07-06 23:43:33.053753801 +0000 UTC m=+43.080659030" Jul 06 23:43:33.068312 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.067592 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b49kq" podStartSLOduration=1.640997158 podStartE2EDuration="8.067574217s" totalImagesPullingTime="6.426577059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:33.066755209 +0000 UTC m=+43.093660439" watchObservedRunningTime="2026-07-06 23:43:33.067574217 +0000 UTC m=+43.094479482" Jul 06 23:43:33.080806 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.080778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:33.081060 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:33.081043 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:43:33.081129 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:33.081108 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls podName:7d410004-c4aa-4f89-9376-6ca83adf6ace nodeName:}" failed. No retries permitted until 2026-07-06 23:43:41.081091662 +0000 UTC m=+51.107996874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls") pod "dns-default-wpbdz" (UID: "7d410004-c4aa-4f89-9376-6ca83adf6ace") : secret "dns-default-metrics-tls" not found Jul 06 23:43:33.081449 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.081430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:33.081596 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:33.081573 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:43:33.081687 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:33.081598 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ccc5f9cc-m6kjg: secret "image-registry-tls" not found Jul 06 23:43:33.081687 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:33.081663 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls podName:1e1fa4c9-5f42-4ee6-89ab-7f735b43977e nodeName:}" failed. No retries permitted until 2026-07-06 23:43:41.081647915 +0000 UTC m=+51.108553127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls") pod "image-registry-76ccc5f9cc-m6kjg" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e") : secret "image-registry-tls" not found Jul 06 23:43:33.085393 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.085347 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" podStartSLOduration=2.26239706 podStartE2EDuration="6.08533016s" totalImagesPullingTime="3.8229331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:33.084486276 +0000 UTC m=+43.111391506" watchObservedRunningTime="2026-07-06 23:43:33.08533016 +0000 UTC m=+43.112235392" Jul 06 23:43:33.147964 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.147931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:33.248782 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.248740 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:33.911091 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.911051 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7hxbm"] Jul 06 23:43:33.933097 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.933064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7hxbm"] Jul 06 23:43:33.933238 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.933198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:33.935206 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.935169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 23:43:33.935323 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.935246 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 23:43:33.935510 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.935496 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qjnpx\"" Jul 06 23:43:33.954260 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.954235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:33.954777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.954757 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w"] Jul 06 23:43:33.989547 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.989495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-crio-socket\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:33.989739 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.989559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-data-volume\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:33.989739 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.989678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:33.989739 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.989721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlppq\" (UniqueName: \"kubernetes.io/projected/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-api-access-nlppq\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:33.989896 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.989757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:33.993830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.993809 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w"] Jul 06 23:43:33.993949 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.993879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:33.999380 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.999072 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 23:43:33.999380 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.999123 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:43:33.999380 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.999152 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 23:43:33.999380 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.999077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 23:43:33.999380 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:33.999073 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r22v6\"" Jul 06 23:43:34.030604 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.030560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:34.052158 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.052136 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:43:34.052619 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.052600 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/0.log" Jul 06 23:43:34.052713 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.052666 2564 generic.go:356] "Generic (PLEG): container finished" podID="1e5f900f-d24d-4c1f-a353-b1f26165a5eb" containerID="9ee8b33aacf29cd5e64cea53273e6e72f1679163823e3088ee971d7909015264" exitCode=255 Jul 06 23:43:34.052794 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.052768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" event={"ID":"1e5f900f-d24d-4c1f-a353-b1f26165a5eb","Type":"ContainerDied","Data":"9ee8b33aacf29cd5e64cea53273e6e72f1679163823e3088ee971d7909015264"} Jul 06 23:43:34.052904 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.052842 2564 scope.go:117] "RemoveContainer" containerID="33d0be19193c2fa25ce7a699e1f9229bb7754ed1c2205243439c08d2904ec1a1" Jul 06 23:43:34.053068 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.053050 2564 scope.go:117] "RemoveContainer" containerID="9ee8b33aacf29cd5e64cea53273e6e72f1679163823e3088ee971d7909015264" Jul 06 23:43:34.053267 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.053240 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sf98j_openshift-console-operator(1e5f900f-d24d-4c1f-a353-b1f26165a5eb)\"" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" podUID="1e5f900f-d24d-4c1f-a353-b1f26165a5eb" Jul 06 23:43:34.090886 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.090856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-crio-socket\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.091045 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.090905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-data-volume\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.091045 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.091045 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlppq\" (UniqueName: \"kubernetes.io/projected/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-api-access-nlppq\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.091233 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzfn\" (UniqueName: \"kubernetes.io/projected/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-kube-api-access-snzfn\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.091233 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091103 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-crio-socket\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.091233 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.091111 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:43:34.091233 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.091233 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.091179 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:34.591159409 +0000 UTC m=+44.618064628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : configmap "kube-rbac-proxy" not found Jul 06 23:43:34.091233 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.091209 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:43:34.091497 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.091253 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:34.591241721 +0000 UTC m=+44.618146929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:43:34.091497 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-config\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.091497 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.091497 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.091362 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-data-volume\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.100955 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.100932 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlppq\" (UniqueName: \"kubernetes.io/projected/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-api-access-nlppq\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.135225 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.135181 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2"] Jul 06 23:43:34.157712 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.157672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2"] Jul 06 23:43:34.157871 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.157807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" Jul 06 23:43:34.160338 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.160315 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tgtxp\"" Jul 06 23:43:34.191924 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.191896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzfn\" (UniqueName: \"kubernetes.io/projected/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-kube-api-access-snzfn\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.192064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.191983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-config\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.192064 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.192035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.193128 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.193102 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-config\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.195523 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.195503 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.201530 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.201504 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzfn\" (UniqueName: \"kubernetes.io/projected/3514de1e-a4bb-4da8-b18c-a8f12dd150bd-kube-api-access-snzfn\") pod \"service-ca-operator-5d46d9ffd4-cdx5w\" (UID: \"3514de1e-a4bb-4da8-b18c-a8f12dd150bd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.293202 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.293163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm42k\" (UniqueName: \"kubernetes.io/projected/619b1b87-4b0d-4860-b470-c687c2960cdf-kube-api-access-vm42k\") pod \"network-check-source-989f8b8dc-h7mt2\" (UID: \"619b1b87-4b0d-4860-b470-c687c2960cdf\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" Jul 06 23:43:34.308133 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.308092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" Jul 06 23:43:34.394175 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.394143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm42k\" (UniqueName: \"kubernetes.io/projected/619b1b87-4b0d-4860-b470-c687c2960cdf-kube-api-access-vm42k\") pod \"network-check-source-989f8b8dc-h7mt2\" (UID: \"619b1b87-4b0d-4860-b470-c687c2960cdf\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" Jul 06 23:43:34.402371 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.402344 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm42k\" (UniqueName: \"kubernetes.io/projected/619b1b87-4b0d-4860-b470-c687c2960cdf-kube-api-access-vm42k\") pod \"network-check-source-989f8b8dc-h7mt2\" (UID: \"619b1b87-4b0d-4860-b470-c687c2960cdf\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" Jul 06 23:43:34.468178 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.468108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" Jul 06 23:43:34.594354 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.594321 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w"] Jul 06 23:43:34.596049 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.596026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.596143 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.596110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:34.596315 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.596290 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:43:34.596421 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.596367 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:35.596348517 +0000 UTC m=+45.623253724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : configmap "kube-rbac-proxy" not found Jul 06 23:43:34.596493 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.596423 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:43:34.596493 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:34.596473 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:35.596461622 +0000 UTC m=+45.623366828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:43:34.597372 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:34.597352 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3514de1e_a4bb_4da8_b18c_a8f12dd150bd.slice/crio-ad852c62fd00490077e17fe676631cd65cd8f0231320b1686e8e28cf8467cae0 WatchSource:0}: Error finding container ad852c62fd00490077e17fe676631cd65cd8f0231320b1686e8e28cf8467cae0: Status 404 returned error can't find the container with id ad852c62fd00490077e17fe676631cd65cd8f0231320b1686e8e28cf8467cae0 Jul 06 23:43:34.637980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.637953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2"] Jul 06 23:43:34.640582 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:34.640554 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619b1b87_4b0d_4860_b470_c687c2960cdf.slice/crio-213116c6ffe65553132ebfc93f659f1dc5f7b2e1c7a430f52c5c1ba2695baf78 WatchSource:0}: Error finding container 213116c6ffe65553132ebfc93f659f1dc5f7b2e1c7a430f52c5c1ba2695baf78: Status 404 returned error can't find the container with id 213116c6ffe65553132ebfc93f659f1dc5f7b2e1c7a430f52c5c1ba2695baf78 Jul 06 23:43:34.901581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:34.901555 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-brd86_9bc12b8c-a240-473d-a57c-b9cd825797ef/dns-node-resolver/0.log" Jul 06 23:43:35.058672 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.058575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:43:35.059114 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.058981 2564 scope.go:117] "RemoveContainer" containerID="9ee8b33aacf29cd5e64cea53273e6e72f1679163823e3088ee971d7909015264" Jul 06 23:43:35.059177 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.059160 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sf98j_openshift-console-operator(1e5f900f-d24d-4c1f-a353-b1f26165a5eb)\"" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" podUID="1e5f900f-d24d-4c1f-a353-b1f26165a5eb" Jul 06 23:43:35.059791 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.059768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" event={"ID":"619b1b87-4b0d-4860-b470-c687c2960cdf","Type":"ContainerStarted","Data":"213116c6ffe65553132ebfc93f659f1dc5f7b2e1c7a430f52c5c1ba2695baf78"} Jul 06 23:43:35.060862 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.060839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" event={"ID":"3514de1e-a4bb-4da8-b18c-a8f12dd150bd","Type":"ContainerStarted","Data":"ad852c62fd00490077e17fe676631cd65cd8f0231320b1686e8e28cf8467cae0"} Jul 06 23:43:35.062348 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.062326 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" event={"ID":"539120a9-c4eb-424e-942b-f7f2047712b7","Type":"ContainerStarted","Data":"fa34d1d679df86007b36ec6148da99afeda939cb2cd926aa60618bd96d46920f"} Jul 06 23:43:35.094013 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.093965 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" podStartSLOduration=1.726736646 podStartE2EDuration="4.093948922s" totalImagesPullingTime="2.367212276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:35.092977096 +0000 UTC m=+45.119882339" watchObservedRunningTime="2026-07-06 23:43:35.093948922 +0000 UTC m=+45.120854151" Jul 06 23:43:35.605324 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.605288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:35.605605 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.605366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:35.605605 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.605429 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:43:35.605605 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.605506 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:37.605483805 +0000 UTC m=+47.632389016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : configmap "kube-rbac-proxy" not found Jul 06 23:43:35.605605 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.605528 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:43:35.605883 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.605621 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:37.605603949 +0000 UTC m=+47.632509161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:43:35.807782 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.807741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:35.808157 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.807832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:35.808157 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.807983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:35.808157 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.808163 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:43.808136481 +0000 UTC m=+53.835041708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : configmap references non-existent config key: service-ca.crt Jul 06 23:43:35.808583 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.808559 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:35.808687 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.808598 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:43:35.808687 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.808641 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls podName:870abde4-501e-4a82-837d-abbb43a831cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:43.808608892 +0000 UTC m=+53.835514110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-84wcc" (UID: "870abde4-501e-4a82-837d-abbb43a831cd") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:35.808687 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.808677 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:43.808659915 +0000 UTC m=+53.835565137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : secret "router-metrics-certs-default" not found Jul 06 23:43:35.902200 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.902129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sd5wt_dbafaea7-f7b6-4dd8-b32a-fd38510ed29d/node-ca/0.log" Jul 06 23:43:35.908409 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:35.908374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:35.908551 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.908513 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:43:35.908599 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:35.908570 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls podName:dc6a3492-579a-4059-8ca1-7023d0b1a28a nodeName:}" failed. No retries permitted until 2026-07-06 23:43:43.908555816 +0000 UTC m=+53.935461024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-f227n" (UID: "dc6a3492-579a-4059-8ca1-7023d0b1a28a") : secret "samples-operator-tls" not found Jul 06 23:43:36.502473 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.502446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b49kq_d5287318-0b55-4737-ac2f-be4aa83b35ba/serve-healthcheck-canary/0.log" Jul 06 23:43:36.628728 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.628696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:36.634884 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.634851 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln"] Jul 06 23:43:36.667967 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.667558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" Jul 06 23:43:36.670379 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.670350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 23:43:36.670519 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.670454 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 23:43:36.670730 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.670707 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k8r6k\"" Jul 06 23:43:36.675665 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.675506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln"] Jul 06 23:43:36.727072 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.727034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:36.816870 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.816834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbbn\" (UniqueName: \"kubernetes.io/projected/050f8482-af52-4f04-aa9c-328646073ab0-kube-api-access-crbbn\") pod \"migrator-d57d58cc9-7xqln\" (UID: \"050f8482-af52-4f04-aa9c-328646073ab0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" Jul 06 23:43:36.827109 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.827079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:36.918377 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.918344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbbn\" (UniqueName: \"kubernetes.io/projected/050f8482-af52-4f04-aa9c-328646073ab0-kube-api-access-crbbn\") pod \"migrator-d57d58cc9-7xqln\" (UID: \"050f8482-af52-4f04-aa9c-328646073ab0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" Jul 06 23:43:36.926450 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.926419 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbbn\" (UniqueName: \"kubernetes.io/projected/050f8482-af52-4f04-aa9c-328646073ab0-kube-api-access-crbbn\") pod \"migrator-d57d58cc9-7xqln\" (UID: \"050f8482-af52-4f04-aa9c-328646073ab0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" Jul 06 23:43:36.984096 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:36.984065 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" Jul 06 23:43:37.625648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:37.625594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:37.626118 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:37.625690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:37.626118 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:37.625723 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:43:37.626118 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:37.625792 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:41.625773903 +0000 UTC m=+51.652679112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : configmap "kube-rbac-proxy" not found Jul 06 23:43:37.626118 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:37.625857 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:43:37.626118 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:37.625917 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:41.625900004 +0000 UTC m=+51.652805217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:43:38.364097 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:38.364056 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:38.364394 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:38.364114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:38.364496 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:38.364479 2564 scope.go:117] "RemoveContainer" containerID="9ee8b33aacf29cd5e64cea53273e6e72f1679163823e3088ee971d7909015264" Jul 06 23:43:38.364674 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:38.364651 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sf98j_openshift-console-operator(1e5f900f-d24d-4c1f-a353-b1f26165a5eb)\"" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" podUID="1e5f900f-d24d-4c1f-a353-b1f26165a5eb" Jul 06 23:43:38.539185 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:38.539155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln"] Jul 06 23:43:38.542491 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:38.542455 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050f8482_af52_4f04_aa9c_328646073ab0.slice/crio-7bc36f0140297c91dcd6ec10f20e3a7db68488278f6b2c0363f0be6f266db5d1 WatchSource:0}: Error finding container 7bc36f0140297c91dcd6ec10f20e3a7db68488278f6b2c0363f0be6f266db5d1: Status 404 returned error can't find the container with id 7bc36f0140297c91dcd6ec10f20e3a7db68488278f6b2c0363f0be6f266db5d1 Jul 06 23:43:39.079864 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:39.079820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" event={"ID":"619b1b87-4b0d-4860-b470-c687c2960cdf","Type":"ContainerStarted","Data":"30864d871fa9ece03c95ec9f0719a11e8bb68b6b638c143382f98425a34bb238"} Jul 06 23:43:39.082100 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:39.082057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" event={"ID":"3514de1e-a4bb-4da8-b18c-a8f12dd150bd","Type":"ContainerStarted","Data":"8e24e872bab4e0f11e0446eabfd6192b383dc5f8024484dad6dc9eb57c88b796"} Jul 06 23:43:39.084045 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:39.084020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" event={"ID":"050f8482-af52-4f04-aa9c-328646073ab0","Type":"ContainerStarted","Data":"7bc36f0140297c91dcd6ec10f20e3a7db68488278f6b2c0363f0be6f266db5d1"} Jul 06 23:43:39.095249 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:39.095190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h7mt2" podStartSLOduration=1.314857666 podStartE2EDuration="5.095174993s" totalImagesPullingTime="3.780317327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:39.093491207 +0000 UTC m=+49.120396439" watchObservedRunningTime="2026-07-06 23:43:39.095174993 +0000 UTC m=+49.122080221" Jul 06 23:43:39.110635 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:39.110584 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" podStartSLOduration=2.296961581 podStartE2EDuration="6.110570925s" totalImagesPullingTime="3.813609344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:39.110519748 +0000 UTC m=+49.137424979" watchObservedRunningTime="2026-07-06 23:43:39.110570925 +0000 UTC m=+49.137476155" Jul 06 23:43:40.187515 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:40.187475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:40.250464 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:40.250411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:40.612581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:40.612542 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:40.713601 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:40.713513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:40.888591 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:40.888546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:40.984314 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:40.984232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:41.093143 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.093100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" event={"ID":"050f8482-af52-4f04-aa9c-328646073ab0","Type":"ContainerStarted","Data":"325cceb908048f676a1fbf943432a6d6b49427358add1c4f389dc63b6a0783cf"} Jul 06 23:43:41.093305 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.093166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" event={"ID":"050f8482-af52-4f04-aa9c-328646073ab0","Type":"ContainerStarted","Data":"f006c8bc78bc137da2ae8f5cafff844343996d8b1ddd8e3d06f0fdb4932d28cd"} Jul 06 23:43:41.109807 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.109758 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7xqln" podStartSLOduration=3.398494371 podStartE2EDuration="5.109742953s" totalImagesPullingTime="1.711248582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:41.108484035 +0000 UTC m=+51.135389264" watchObservedRunningTime="2026-07-06 23:43:41.109742953 +0000 UTC m=+51.136648182" Jul 06 23:43:41.157573 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.157535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:41.157785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.157695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:41.157785 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:41.157742 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:43:41.157917 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:41.157797 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:43:41.157917 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:41.157807 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ccc5f9cc-m6kjg: secret "image-registry-tls" not found Jul 06 23:43:41.157917 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:41.157823 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls podName:7d410004-c4aa-4f89-9376-6ca83adf6ace nodeName:}" failed. No retries permitted until 2026-07-06 23:43:57.157800403 +0000 UTC m=+67.184705615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls") pod "dns-default-wpbdz" (UID: "7d410004-c4aa-4f89-9376-6ca83adf6ace") : secret "dns-default-metrics-tls" not found Jul 06 23:43:41.157917 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:41.157842 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls podName:1e1fa4c9-5f42-4ee6-89ab-7f735b43977e nodeName:}" failed. No retries permitted until 2026-07-06 23:43:57.157831999 +0000 UTC m=+67.184737207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls") pod "image-registry-76ccc5f9cc-m6kjg" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e") : secret "image-registry-tls" not found Jul 06 23:43:41.183537 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.183507 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:41.588221 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.588181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:41.661588 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.661542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:41.661796 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.661613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:41.661796 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:41.661753 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:43:41.661871 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:41.661815 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls podName:7dc94e3c-1c5b-4495-80d4-ff4288bdedd2 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:49.66179958 +0000 UTC m=+59.688704787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hxbm" (UID: "7dc94e3c-1c5b-4495-80d4-ff4288bdedd2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:43:41.662113 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:41.662096 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:42.479534 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.479498 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-rdtrf"] Jul 06 23:43:42.493402 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.493374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-rdtrf"] Jul 06 23:43:42.493568 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.493472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.495825 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.495798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 23:43:42.495825 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.495818 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 23:43:42.496093 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.495811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 23:43:42.496093 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.495909 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 23:43:42.496742 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.496729 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2mc25\"" Jul 06 23:43:42.511120 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.511090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:42.569697 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.569625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aac324ea-a3e8-4afd-b9dd-48e765eb2440-signing-key\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.569881 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.569800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8kx2\" (UniqueName: \"kubernetes.io/projected/aac324ea-a3e8-4afd-b9dd-48e765eb2440-kube-api-access-q8kx2\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.569881 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.569836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aac324ea-a3e8-4afd-b9dd-48e765eb2440-signing-cabundle\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.612404 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.612361 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:42.670970 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.670935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aac324ea-a3e8-4afd-b9dd-48e765eb2440-signing-key\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.671133 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.670997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8kx2\" (UniqueName: \"kubernetes.io/projected/aac324ea-a3e8-4afd-b9dd-48e765eb2440-kube-api-access-q8kx2\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.671133 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.671021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aac324ea-a3e8-4afd-b9dd-48e765eb2440-signing-cabundle\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.671724 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.671706 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aac324ea-a3e8-4afd-b9dd-48e765eb2440-signing-cabundle\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.673253 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.673235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aac324ea-a3e8-4afd-b9dd-48e765eb2440-signing-key\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.679762 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.679739 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8kx2\" (UniqueName: \"kubernetes.io/projected/aac324ea-a3e8-4afd-b9dd-48e765eb2440-kube-api-access-q8kx2\") pod \"service-ca-758f599c5-rdtrf\" (UID: \"aac324ea-a3e8-4afd-b9dd-48e765eb2440\") " pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.802498 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.802457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-rdtrf" Jul 06 23:43:42.940306 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:42.940275 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-rdtrf"] Jul 06 23:43:42.943304 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:42.943276 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac324ea_a3e8_4afd_b9dd_48e765eb2440.slice/crio-70ed560cb1c0a78ae52808689691ccb1bdab86f4950e269979f5ccaa5606c1b4 WatchSource:0}: Error finding container 70ed560cb1c0a78ae52808689691ccb1bdab86f4950e269979f5ccaa5606c1b4: Status 404 returned error can't find the container with id 70ed560cb1c0a78ae52808689691ccb1bdab86f4950e269979f5ccaa5606c1b4 Jul 06 23:43:43.101882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.101785 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-rdtrf" event={"ID":"aac324ea-a3e8-4afd-b9dd-48e765eb2440","Type":"ContainerStarted","Data":"270efd5e9854f8c7949e87d7de365eb411f5623c636f142bcf5667a32cc284e2"} Jul 06 23:43:43.101882 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.101837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-rdtrf" event={"ID":"aac324ea-a3e8-4afd-b9dd-48e765eb2440","Type":"ContainerStarted","Data":"70ed560cb1c0a78ae52808689691ccb1bdab86f4950e269979f5ccaa5606c1b4"} Jul 06 23:43:43.125150 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.122573 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-rdtrf" podStartSLOduration=1.122550998 podStartE2EDuration="1.122550998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:43.120857294 +0000 UTC m=+53.147762524" watchObservedRunningTime="2026-07-06 23:43:43.122550998 +0000 UTC m=+53.149456229" Jul 06 23:43:43.433224 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.433131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:43.884502 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.884466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:43.884978 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.884552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:43.884978 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.884604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:43.884978 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:43.884663 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:59.884622744 +0000 UTC m=+69.911527971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : configmap references non-existent config key: service-ca.crt Jul 06 23:43:43.884978 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:43.884733 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:43.884978 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:43.884790 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls podName:870abde4-501e-4a82-837d-abbb43a831cd nodeName:}" failed. No retries permitted until 2026-07-06 23:43:59.884775885 +0000 UTC m=+69.911681092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-84wcc" (UID: "870abde4-501e-4a82-837d-abbb43a831cd") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:43:43.884978 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:43.884863 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:43:43.884978 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:43.884921 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs podName:0522ae39-76b7-493c-9a01-2913bf880b08 nodeName:}" failed. No retries permitted until 2026-07-06 23:43:59.884905606 +0000 UTC m=+69.911810825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs") pod "router-default-7d86c67597-d9q6b" (UID: "0522ae39-76b7-493c-9a01-2913bf880b08") : secret "router-metrics-certs-default" not found Jul 06 23:43:43.974643 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.974586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:43.985426 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:43.985394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:43:43.985592 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:43.985559 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:43:43.985666 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:43:43.985643 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls podName:dc6a3492-579a-4059-8ca1-7023d0b1a28a nodeName:}" failed. No retries permitted until 2026-07-06 23:43:59.985611019 +0000 UTC m=+70.012516239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-f227n" (UID: "dc6a3492-579a-4059-8ca1-7023d0b1a28a") : secret "samples-operator-tls" not found Jul 06 23:43:44.089222 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:44.089189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:44.185317 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:44.185223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:44.283578 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:44.283539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:44.383720 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:44.383687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:45.076080 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:45.076045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:45.482825 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:45.482735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:45.687303 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:45.687260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:45.870087 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:45.870054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:46.125373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:46.125291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:46.877657 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:46.877591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:47.074943 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:47.074909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:47.479098 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:47.479058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:47.677096 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:47.677051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:47.778229 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:47.778195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:47.999575 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:47.999544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:48.297854 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:48.297820 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:48.676750 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:48.676617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:48.732768 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:48.732737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:48.919261 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:48.919221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:48.981858 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:48.981777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:49.077255 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.077222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:49.272463 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.272370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:49.649429 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.649395 2564 scope.go:117] "RemoveContainer" containerID="9ee8b33aacf29cd5e64cea53273e6e72f1679163823e3088ee971d7909015264" Jul 06 23:43:49.740250 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.740211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:49.742602 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.742581 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dc94e3c-1c5b-4495-80d4-ff4288bdedd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hxbm\" (UID: \"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2\") " pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:49.842240 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.842205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7hxbm" Jul 06 23:43:49.878259 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.878211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:49.984384 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:49.984356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7hxbm"] Jul 06 23:43:49.988119 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:49.988094 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc94e3c_1c5b_4495_80d4_ff4288bdedd2.slice/crio-4a6aa2e75d428aa53bded6d9b4f62fe57a547b31ae34a2d5ea8c9301f65df907 WatchSource:0}: Error finding container 4a6aa2e75d428aa53bded6d9b4f62fe57a547b31ae34a2d5ea8c9301f65df907: Status 404 returned error can't find the container with id 4a6aa2e75d428aa53bded6d9b4f62fe57a547b31ae34a2d5ea8c9301f65df907 Jul 06 23:43:50.127551 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.127521 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:43:50.127748 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.127607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" event={"ID":"1e5f900f-d24d-4c1f-a353-b1f26165a5eb","Type":"ContainerStarted","Data":"97d8bf1750e3b5c3903926fc6f9b36d78262f1327721d73d6928a63c6ffeb1d4"} Jul 06 23:43:50.127959 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.127912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:50.128938 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.128914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hxbm" event={"ID":"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2","Type":"ContainerStarted","Data":"4a6aa2e75d428aa53bded6d9b4f62fe57a547b31ae34a2d5ea8c9301f65df907"} Jul 06 23:43:50.146853 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.146807 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" podStartSLOduration=19.426418362 podStartE2EDuration="23.146792941s" totalImagesPullingTime="3.720374579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:50.145793168 +0000 UTC m=+60.172698397" watchObservedRunningTime="2026-07-06 23:43:50.146792941 +0000 UTC m=+60.173698148" Jul 06 23:43:50.320813 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.320767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:50.479767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.479730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:50.671348 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.670722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:50.864007 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:50.863975 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-sf98j" Jul 06 23:43:51.054148 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.054093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:51.154649 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.154578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:51.253136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.252404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:51.353136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.352855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:51.455948 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.455913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:51.672533 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.672449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:51.773312 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.773276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:51.991183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:51.991107 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9pgh" Jul 06 23:43:52.275620 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:52.275585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:52.548483 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:52.548396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:52.652497 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:52.652462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:52.754497 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:52.754462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:52.876914 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:52.876818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:52.976664 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:52.976611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:53.029254 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:53.029224 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:53.089500 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:53.089467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:53.143091 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:53.143018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:53.489073 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:53.488987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:53.730910 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:53.730860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:54.078522 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:54.078498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:54.146157 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:54.146128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hxbm" event={"ID":"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2","Type":"ContainerStarted","Data":"7df61849f978aa71ee1a7ab3b8c250c64511095a6fa7407d41cb2a9c110fa6a9"} Jul 06 23:43:54.570713 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:54.570684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:54.677597 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:54.677568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:54.884814 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:54.884707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:55.081231 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:55.081197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:55.151432 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:55.151346 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hxbm" event={"ID":"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2","Type":"ContainerStarted","Data":"e1ff8c2cf4be3073759567de36c5466cd03d04f74557b22500bb7a99cf916954"} Jul 06 23:43:55.361323 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:55.361285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:55.457790 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:55.457713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:43:56.308375 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.308341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:56.308836 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.308442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:56.311125 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.311099 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:43:56.311240 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.311221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:43:56.321354 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.321329 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670307dd-877f-4f25-ad8c-3b6bb6ce1134-metrics-certs\") pod \"network-metrics-daemon-6lqdv\" (UID: \"670307dd-877f-4f25-ad8c-3b6bb6ce1134\") " pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:56.321471 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.321329 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd-original-pull-secret\") pod \"global-pull-secret-syncer-7rs8n\" (UID: \"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd\") " pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:56.360847 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.360813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rs8n" Jul 06 23:43:56.375542 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.375515 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xv6vb\"" Jul 06 23:43:56.383605 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.383581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lqdv" Jul 06 23:43:56.409255 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.409228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:56.412090 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.412062 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khz9t\" (UniqueName: \"kubernetes.io/projected/d4804710-463b-4d7e-b473-bc65adf9d9ae-kube-api-access-khz9t\") pod \"network-check-target-jj75g\" (UID: \"d4804710-463b-4d7e-b473-bc65adf9d9ae\") " pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:56.670142 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.670069 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2m52g\"" Jul 06 23:43:56.677884 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:56.677857 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:57.216662 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.215007 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jj75g"] Jul 06 23:43:57.220743 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.219795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:57.220743 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.219921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:57.231141 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.231099 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d410004-c4aa-4f89-9376-6ca83adf6ace-metrics-tls\") pod \"dns-default-wpbdz\" (UID: \"7d410004-c4aa-4f89-9376-6ca83adf6ace\") " pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:57.231378 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:57.231348 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4804710_463b_4d7e_b473_bc65adf9d9ae.slice/crio-beb8977216c0915b4104f4a493b667b809e94dc8bf629d648aeac508094497c4 WatchSource:0}: Error finding container beb8977216c0915b4104f4a493b667b809e94dc8bf629d648aeac508094497c4: Status 404 returned error can't find the container with id beb8977216c0915b4104f4a493b667b809e94dc8bf629d648aeac508094497c4 Jul 06 23:43:57.233063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.233039 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"image-registry-76ccc5f9cc-m6kjg\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:57.241533 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.241507 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6lqdv"] Jul 06 23:43:57.253906 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.253883 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7rs8n"] Jul 06 23:43:57.411776 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:57.411711 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670307dd_877f_4f25_ad8c_3b6bb6ce1134.slice/crio-dd3d4323c0a87309cda08e1e0ff5e643b3f3645938fdd1822f4d0cb8c5927282 WatchSource:0}: Error finding container dd3d4323c0a87309cda08e1e0ff5e643b3f3645938fdd1822f4d0cb8c5927282: Status 404 returned error can't find the container with id dd3d4323c0a87309cda08e1e0ff5e643b3f3645938fdd1822f4d0cb8c5927282 Jul 06 23:43:57.412923 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:57.412896 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3c82fa_4033_4a51_aeb3_3fb8441ac7cd.slice/crio-9a4683c770364caff2bd8a276c17507ee6eb29108bb196e832603439eee2f5ec WatchSource:0}: Error finding container 9a4683c770364caff2bd8a276c17507ee6eb29108bb196e832603439eee2f5ec: Status 404 returned error can't find the container with id 9a4683c770364caff2bd8a276c17507ee6eb29108bb196e832603439eee2f5ec Jul 06 23:43:57.461929 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.461911 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lx775\"" Jul 06 23:43:57.470710 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.470686 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:57.476974 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.476958 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6mrd\"" Jul 06 23:43:57.485054 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.485034 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wpbdz" Jul 06 23:43:57.651563 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.651529 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76ccc5f9cc-m6kjg"] Jul 06 23:43:57.655684 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:57.655622 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1fa4c9_5f42_4ee6_89ab_7f735b43977e.slice/crio-431a665e305f670321106421c1523db8bdb66c42132e48158386f5c41b3b52ce WatchSource:0}: Error finding container 431a665e305f670321106421c1523db8bdb66c42132e48158386f5c41b3b52ce: Status 404 returned error can't find the container with id 431a665e305f670321106421c1523db8bdb66c42132e48158386f5c41b3b52ce Jul 06 23:43:57.659589 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:57.659567 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wpbdz"] Jul 06 23:43:57.662201 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:43:57.662156 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d410004_c4aa_4f89_9376_6ca83adf6ace.slice/crio-f5234c2a78bae5c458f50d4612b32437f2ff9eb6a5c05f0ca032ff72787131cb WatchSource:0}: Error finding container f5234c2a78bae5c458f50d4612b32437f2ff9eb6a5c05f0ca032ff72787131cb: Status 404 returned error can't find the container with id f5234c2a78bae5c458f50d4612b32437f2ff9eb6a5c05f0ca032ff72787131cb Jul 06 23:43:58.168549 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.168509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wpbdz" event={"ID":"7d410004-c4aa-4f89-9376-6ca83adf6ace","Type":"ContainerStarted","Data":"f5234c2a78bae5c458f50d4612b32437f2ff9eb6a5c05f0ca032ff72787131cb"} Jul 06 23:43:58.170389 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.170323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7rs8n" event={"ID":"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd","Type":"ContainerStarted","Data":"9a4683c770364caff2bd8a276c17507ee6eb29108bb196e832603439eee2f5ec"} Jul 06 23:43:58.174696 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.174616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hxbm" event={"ID":"7dc94e3c-1c5b-4495-80d4-ff4288bdedd2","Type":"ContainerStarted","Data":"cdd840dcff332363ce15c16c4fbd1537f2d565c7d771d41eac55984fe665ee10"} Jul 06 23:43:58.176563 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.176536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" event={"ID":"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e","Type":"ContainerStarted","Data":"4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6"} Jul 06 23:43:58.176690 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.176583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" event={"ID":"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e","Type":"ContainerStarted","Data":"431a665e305f670321106421c1523db8bdb66c42132e48158386f5c41b3b52ce"} Jul 06 23:43:58.176807 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.176790 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:43:58.179034 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.178984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6lqdv" event={"ID":"670307dd-877f-4f25-ad8c-3b6bb6ce1134","Type":"ContainerStarted","Data":"dd3d4323c0a87309cda08e1e0ff5e643b3f3645938fdd1822f4d0cb8c5927282"} Jul 06 23:43:58.180881 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.180816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jj75g" event={"ID":"d4804710-463b-4d7e-b473-bc65adf9d9ae","Type":"ContainerStarted","Data":"0fdfaf19ec1984459be3ef3edca2819c52dc9d10256caf2fc09ce0e0b5c1bf93"} Jul 06 23:43:58.180881 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.180854 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jj75g" event={"ID":"d4804710-463b-4d7e-b473-bc65adf9d9ae","Type":"ContainerStarted","Data":"beb8977216c0915b4104f4a493b667b809e94dc8bf629d648aeac508094497c4"} Jul 06 23:43:58.181038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.180908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:43:58.210443 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.210374 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" podStartSLOduration=45.21035652 podStartE2EDuration="45.21035652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:58.20999662 +0000 UTC m=+68.236901849" watchObservedRunningTime="2026-07-06 23:43:58.21035652 +0000 UTC m=+68.237261751" Jul 06 23:43:58.210612 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.210572 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7hxbm" podStartSLOduration=17.856984706 podStartE2EDuration="25.210561626s" totalImagesPullingTime="7.35357692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:43:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:58.191553878 +0000 UTC m=+68.218459107" watchObservedRunningTime="2026-07-06 23:43:58.210561626 +0000 UTC m=+68.237466856" Jul 06 23:43:58.224501 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:58.224447 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jj75g" podStartSLOduration=68.224427996 podStartE2EDuration="1m8.224427996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:58.223307438 +0000 UTC m=+68.250212666" watchObservedRunningTime="2026-07-06 23:43:58.224427996 +0000 UTC m=+68.251333225" Jul 06 23:43:59.187094 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.187000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6lqdv" event={"ID":"670307dd-877f-4f25-ad8c-3b6bb6ce1134","Type":"ContainerStarted","Data":"67912b14857c47dc3f159185501a642b4a3f81aec44872455f66d93499673f3f"} Jul 06 23:43:59.187094 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.187070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6lqdv" event={"ID":"670307dd-877f-4f25-ad8c-3b6bb6ce1134","Type":"ContainerStarted","Data":"13cb1373ed1a35e2cdb83c327a679b5544c64607e32eb7109bf6d57d816408ef"} Jul 06 23:43:59.205517 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.205461 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6lqdv" podStartSLOduration=67.791507311 podStartE2EDuration="1m9.205442544s" totalImagesPullingTime="1.413935233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:43:59.205099583 +0000 UTC m=+69.232004813" watchObservedRunningTime="2026-07-06 23:43:59.205442544 +0000 UTC m=+69.232347775" Jul 06 23:43:59.943025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.942941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:59.943167 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.943131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:43:59.943167 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.943159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:59.943800 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.943765 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0522ae39-76b7-493c-9a01-2913bf880b08-service-ca-bundle\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:59.945719 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.945692 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0522ae39-76b7-493c-9a01-2913bf880b08-metrics-certs\") pod \"router-default-7d86c67597-d9q6b\" (UID: \"0522ae39-76b7-493c-9a01-2913bf880b08\") " pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:43:59.945843 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:43:59.945818 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/870abde4-501e-4a82-837d-abbb43a831cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-84wcc\" (UID: \"870abde4-501e-4a82-837d-abbb43a831cd\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:44:00.044396 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.044349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:44:00.047051 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.047027 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6a3492-579a-4059-8ca1-7023d0b1a28a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-f227n\" (UID: \"dc6a3492-579a-4059-8ca1-7023d0b1a28a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:44:00.149085 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.149050 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8hw9l\"" Jul 06 23:44:00.157037 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.157010 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" Jul 06 23:44:00.157933 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.157911 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8z6zv\"" Jul 06 23:44:00.166927 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.166904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:44:00.193893 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.193810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wpbdz" event={"ID":"7d410004-c4aa-4f89-9376-6ca83adf6ace","Type":"ContainerStarted","Data":"ca346628d62a644793f6e47a767cccfc915565ee25dac961ea8c9b89d037f98b"} Jul 06 23:44:00.193893 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.193871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wpbdz" event={"ID":"7d410004-c4aa-4f89-9376-6ca83adf6ace","Type":"ContainerStarted","Data":"90ef8ed36db5912e7a85d1a0e6a13edf24f6c9ae2f2605b966de96258c9f36c8"} Jul 06 23:44:00.194361 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.194051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wpbdz" Jul 06 23:44:00.213418 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.213357 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wpbdz" podStartSLOduration=33.199631293 podStartE2EDuration="35.213338695s" totalImagesPullingTime="2.013707402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:00.21204015 +0000 UTC m=+70.238945381" watchObservedRunningTime="2026-07-06 23:44:00.213338695 +0000 UTC m=+70.240243926" Jul 06 23:44:00.229320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.229296 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vg7xt\"" Jul 06 23:44:00.237219 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:00.237192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" Jul 06 23:44:01.615261 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.614180 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc"] Jul 06 23:44:01.624065 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:01.624032 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870abde4_501e_4a82_837d_abbb43a831cd.slice/crio-f3a50e2f5eed0a39a1f8f18d4492a918904b50feadcee398a0960ff752b193e2 WatchSource:0}: Error finding container f3a50e2f5eed0a39a1f8f18d4492a918904b50feadcee398a0960ff752b193e2: Status 404 returned error can't find the container with id f3a50e2f5eed0a39a1f8f18d4492a918904b50feadcee398a0960ff752b193e2 Jul 06 23:44:01.726844 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.726753 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt"] Jul 06 23:44:01.730165 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.730140 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt"] Jul 06 23:44:01.730307 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.730265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.732806 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.732784 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 23:44:01.733219 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733195 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 23:44:01.733325 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733201 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm"] Jul 06 23:44:01.733325 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:01.733847 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 23:44:01.733847 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733694 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 23:44:01.733847 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733699 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 23:44:01.733847 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733724 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 23:44:01.733847 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.733764 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 23:44:01.735243 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.735218 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tlkcb\"" Jul 06 23:44:01.735383 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.735367 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 23:44:01.736487 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.736470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.738407 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.738327 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 23:44:01.742382 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.742359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm"] Jul 06 23:44:01.743878 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.743858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt"] Jul 06 23:44:01.745380 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.745357 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt"] Jul 06 23:44:01.753965 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.753905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-hub\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52c92b8b-0678-4c12-8737-4162ebb2f65e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwql7\" (UniqueName: \"kubernetes.io/projected/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-kube-api-access-fwql7\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-ca\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wbg\" (UniqueName: \"kubernetes.io/projected/52c92b8b-0678-4c12-8737-4162ebb2f65e-kube-api-access-x9wbg\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0af8243-7525-4bca-9796-3f51c03a39e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-675cb94cb4-6jqlt\" (UID: \"e0af8243-7525-4bca-9796-3f51c03a39e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ssb\" (UniqueName: \"kubernetes.io/projected/e0af8243-7525-4bca-9796-3f51c03a39e5-kube-api-access-l7ssb\") pod \"managed-serviceaccount-addon-agent-675cb94cb4-6jqlt\" (UID: \"e0af8243-7525-4bca-9796-3f51c03a39e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52c92b8b-0678-4c12-8737-4162ebb2f65e-tmp\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.758980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.758802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-additional-service-ca\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.820579 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.820533 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-bjj78"] Jul 06 23:44:01.824042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.824018 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv"] Jul 06 23:44:01.824180 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.824158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-bjj78" Jul 06 23:44:01.826894 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.826874 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 23:44:01.827058 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.826872 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-plsrs\"" Jul 06 23:44:01.827303 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.827283 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 23:44:01.827662 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.827606 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:01.829878 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.829834 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 23:44:01.830007 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.829987 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d7mjb\"" Jul 06 23:44:01.830267 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.830254 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 23:44:01.836042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.836023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv"] Jul 06 23:44:01.837945 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.837925 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-bjj78"] Jul 06 23:44:01.851041 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.851021 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n"] Jul 06 23:44:01.852557 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.852534 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d86c67597-d9q6b"] Jul 06 23:44:01.854739 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:01.854712 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0522ae39_76b7_493c_9a01_2913bf880b08.slice/crio-5cdfaa5703a1894b80272887cae1295f695cb6127d45652bc2c31f525dd44afc WatchSource:0}: Error finding container 5cdfaa5703a1894b80272887cae1295f695cb6127d45652bc2c31f525dd44afc: Status 404 returned error can't find the container with id 5cdfaa5703a1894b80272887cae1295f695cb6127d45652bc2c31f525dd44afc Jul 06 23:44:01.859785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.859766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.859888 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.859804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686n7\" (UniqueName: \"kubernetes.io/projected/bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1-kube-api-access-686n7\") pod \"downloads-685bfb5f64-bjj78\" (UID: \"bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1\") " pod="openshift-console/downloads-685bfb5f64-bjj78" Jul 06 23:44:01.859888 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.859833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-additional-service-ca\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.859996 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.859887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-hub\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.859996 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.859930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.859996 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.859955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52c92b8b-0678-4c12-8737-4162ebb2f65e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.860402 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwql7\" (UniqueName: \"kubernetes.io/projected/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-kube-api-access-fwql7\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.860530 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-ca\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.860530 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wbg\" (UniqueName: \"kubernetes.io/projected/52c92b8b-0678-4c12-8737-4162ebb2f65e-kube-api-access-x9wbg\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.860530 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0af8243-7525-4bca-9796-3f51c03a39e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-675cb94cb4-6jqlt\" (UID: \"e0af8243-7525-4bca-9796-3f51c03a39e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:01.860733 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b775ca7-c969-4e1c-a836-bcdaa9803f68-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6rmbv\" (UID: \"7b775ca7-c969-4e1c-a836-bcdaa9803f68\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:01.860733 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ssb\" (UniqueName: \"kubernetes.io/projected/e0af8243-7525-4bca-9796-3f51c03a39e5-kube-api-access-l7ssb\") pod \"managed-serviceaccount-addon-agent-675cb94cb4-6jqlt\" (UID: \"e0af8243-7525-4bca-9796-3f51c03a39e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:01.860733 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52c92b8b-0678-4c12-8737-4162ebb2f65e-tmp\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.860733 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860687 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b775ca7-c969-4e1c-a836-bcdaa9803f68-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6rmbv\" (UID: \"7b775ca7-c969-4e1c-a836-bcdaa9803f68\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:01.861000 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:01.861066 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.860512 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-additional-service-ca\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.861757 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.861732 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52c92b8b-0678-4c12-8737-4162ebb2f65e-tmp\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.862807 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.862773 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.862807 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.862780 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-hub\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.863324 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.863305 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52c92b8b-0678-4c12-8737-4162ebb2f65e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.863376 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.863343 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.863615 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.863595 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-ca\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.864046 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.864031 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0af8243-7525-4bca-9796-3f51c03a39e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-675cb94cb4-6jqlt\" (UID: \"e0af8243-7525-4bca-9796-3f51c03a39e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:01.872164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.872120 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wbg\" (UniqueName: \"kubernetes.io/projected/52c92b8b-0678-4c12-8737-4162ebb2f65e-kube-api-access-x9wbg\") pod \"klusterlet-addon-workmgr-5b67bb96d7-s4qgm\" (UID: \"52c92b8b-0678-4c12-8737-4162ebb2f65e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:01.877876 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.877850 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ssb\" (UniqueName: \"kubernetes.io/projected/e0af8243-7525-4bca-9796-3f51c03a39e5-kube-api-access-l7ssb\") pod \"managed-serviceaccount-addon-agent-675cb94cb4-6jqlt\" (UID: \"e0af8243-7525-4bca-9796-3f51c03a39e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:01.878785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.878764 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwql7\" (UniqueName: \"kubernetes.io/projected/586fd041-ba8d-4941-b9c5-39c9e2ae4d59-kube-api-access-fwql7\") pod \"cluster-proxy-proxy-agent-697b8dbc9c-g9bpt\" (UID: \"586fd041-ba8d-4941-b9c5-39c9e2ae4d59\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:01.941646 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.941584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:01.962008 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.961980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b775ca7-c969-4e1c-a836-bcdaa9803f68-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6rmbv\" (UID: \"7b775ca7-c969-4e1c-a836-bcdaa9803f68\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:01.962377 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.962027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686n7\" (UniqueName: \"kubernetes.io/projected/bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1-kube-api-access-686n7\") pod \"downloads-685bfb5f64-bjj78\" (UID: \"bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1\") " pod="openshift-console/downloads-685bfb5f64-bjj78" Jul 06 23:44:01.962377 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.962090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b775ca7-c969-4e1c-a836-bcdaa9803f68-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6rmbv\" (UID: \"7b775ca7-c969-4e1c-a836-bcdaa9803f68\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:01.962774 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.962756 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b775ca7-c969-4e1c-a836-bcdaa9803f68-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6rmbv\" (UID: \"7b775ca7-c969-4e1c-a836-bcdaa9803f68\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:01.964375 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.964357 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b775ca7-c969-4e1c-a836-bcdaa9803f68-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6rmbv\" (UID: \"7b775ca7-c969-4e1c-a836-bcdaa9803f68\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:01.971519 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:01.971497 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686n7\" (UniqueName: \"kubernetes.io/projected/bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1-kube-api-access-686n7\") pod \"downloads-685bfb5f64-bjj78\" (UID: \"bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1\") " pod="openshift-console/downloads-685bfb5f64-bjj78" Jul 06 23:44:02.026415 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.026377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:02.049581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.049555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" Jul 06 23:44:02.059642 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.059609 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" Jul 06 23:44:02.078868 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.078842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:02.139219 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.138895 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-bjj78" Jul 06 23:44:02.145090 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.145024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" Jul 06 23:44:02.214902 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.214753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" event={"ID":"870abde4-501e-4a82-837d-abbb43a831cd","Type":"ContainerStarted","Data":"f3a50e2f5eed0a39a1f8f18d4492a918904b50feadcee398a0960ff752b193e2"} Jul 06 23:44:02.218072 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.217546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d86c67597-d9q6b" event={"ID":"0522ae39-76b7-493c-9a01-2913bf880b08","Type":"ContainerStarted","Data":"a2fa38a72eb89a5b24426f0dbdafc9fa6f3a67acb343e8ca5dcfd180c332a62a"} Jul 06 23:44:02.218072 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.217646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d86c67597-d9q6b" event={"ID":"0522ae39-76b7-493c-9a01-2913bf880b08","Type":"ContainerStarted","Data":"5cdfaa5703a1894b80272887cae1295f695cb6127d45652bc2c31f525dd44afc"} Jul 06 23:44:02.219608 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.219563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" event={"ID":"dc6a3492-579a-4059-8ca1-7023d0b1a28a","Type":"ContainerStarted","Data":"771f28567906b5180e8e2fc92ee3137ba64570ce682b5745cd82908734fe17f8"} Jul 06 23:44:02.230730 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.230687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt"] Jul 06 23:44:02.240554 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.239322 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7d86c67597-d9q6b" podStartSLOduration=35.239299926 podStartE2EDuration="35.239299926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:02.239242446 +0000 UTC m=+72.266147677" watchObservedRunningTime="2026-07-06 23:44:02.239299926 +0000 UTC m=+72.266205155" Jul 06 23:44:02.240554 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.239778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7rs8n" event={"ID":"fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd","Type":"ContainerStarted","Data":"a2bc85a4af6f8754af0cf15bb2f88fc163cbad3df6de5bd3994a3937caf9cd07"} Jul 06 23:44:02.243236 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:02.243184 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod586fd041_ba8d_4941_b9c5_39c9e2ae4d59.slice/crio-27068157851cfbfeb3927ecdd4e275b6bca94a27b5871cc62b3f4f1c176a445a WatchSource:0}: Error finding container 27068157851cfbfeb3927ecdd4e275b6bca94a27b5871cc62b3f4f1c176a445a: Status 404 returned error can't find the container with id 27068157851cfbfeb3927ecdd4e275b6bca94a27b5871cc62b3f4f1c176a445a Jul 06 23:44:02.261557 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.259434 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7rs8n" podStartSLOduration=68.182036417 podStartE2EDuration="1m12.259413551s" totalImagesPullingTime="4.077377134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:02.257114059 +0000 UTC m=+72.284019279" watchObservedRunningTime="2026-07-06 23:44:02.259413551 +0000 UTC m=+72.286318780" Jul 06 23:44:02.265116 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.264555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt"] Jul 06 23:44:02.266833 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:02.266783 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0af8243_7525_4bca_9796_3f51c03a39e5.slice/crio-6781ae62d42c66a56bbb2e2c399b61e5d8a6ed53ed43a332784d0912f604a680 WatchSource:0}: Error finding container 6781ae62d42c66a56bbb2e2c399b61e5d8a6ed53ed43a332784d0912f604a680: Status 404 returned error can't find the container with id 6781ae62d42c66a56bbb2e2c399b61e5d8a6ed53ed43a332784d0912f604a680 Jul 06 23:44:02.284210 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.284179 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm"] Jul 06 23:44:02.285011 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:02.284974 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c92b8b_0678_4c12_8737_4162ebb2f65e.slice/crio-e251825a02013f6a9156003a97a9d85d613b5823e75757af89a92219c1889b52 WatchSource:0}: Error finding container e251825a02013f6a9156003a97a9d85d613b5823e75757af89a92219c1889b52: Status 404 returned error can't find the container with id e251825a02013f6a9156003a97a9d85d613b5823e75757af89a92219c1889b52 Jul 06 23:44:02.338265 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.338237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-bjj78"] Jul 06 23:44:02.340257 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:02.340230 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbec8d32_f9bf_44fe_b4ec_b0b35dfff7d1.slice/crio-4050a64af6fdee8e0a8e139155a4f89c179b2c50b165c867d5f61ee3246691a3 WatchSource:0}: Error finding container 4050a64af6fdee8e0a8e139155a4f89c179b2c50b165c867d5f61ee3246691a3: Status 404 returned error can't find the container with id 4050a64af6fdee8e0a8e139155a4f89c179b2c50b165c867d5f61ee3246691a3 Jul 06 23:44:02.353831 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:02.353807 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv"] Jul 06 23:44:02.356519 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:02.356495 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b775ca7_c969_4e1c_a836_bcdaa9803f68.slice/crio-e46a8fd66aaca1efd34388c38096a29ce65ea6f2d4eb975508ce03be1efb96d5 WatchSource:0}: Error finding container e46a8fd66aaca1efd34388c38096a29ce65ea6f2d4eb975508ce03be1efb96d5: Status 404 returned error can't find the container with id e46a8fd66aaca1efd34388c38096a29ce65ea6f2d4eb975508ce03be1efb96d5 Jul 06 23:44:03.167988 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.167910 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:44:03.172503 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.172246 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:44:03.253158 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.253122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" event={"ID":"52c92b8b-0678-4c12-8737-4162ebb2f65e","Type":"ContainerStarted","Data":"e251825a02013f6a9156003a97a9d85d613b5823e75757af89a92219c1889b52"} Jul 06 23:44:03.254723 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.254655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-bjj78" event={"ID":"bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1","Type":"ContainerStarted","Data":"4050a64af6fdee8e0a8e139155a4f89c179b2c50b165c867d5f61ee3246691a3"} Jul 06 23:44:03.258404 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.258374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" event={"ID":"586fd041-ba8d-4941-b9c5-39c9e2ae4d59","Type":"ContainerStarted","Data":"27068157851cfbfeb3927ecdd4e275b6bca94a27b5871cc62b3f4f1c176a445a"} Jul 06 23:44:03.260530 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.260466 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" event={"ID":"7b775ca7-c969-4e1c-a836-bcdaa9803f68","Type":"ContainerStarted","Data":"e46a8fd66aaca1efd34388c38096a29ce65ea6f2d4eb975508ce03be1efb96d5"} Jul 06 23:44:03.265667 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.265130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" event={"ID":"e0af8243-7525-4bca-9796-3f51c03a39e5","Type":"ContainerStarted","Data":"6781ae62d42c66a56bbb2e2c399b61e5d8a6ed53ed43a332784d0912f604a680"} Jul 06 23:44:03.265667 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.265174 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:44:03.268236 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.268045 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d86c67597-d9q6b" Jul 06 23:44:03.701638 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.701495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:03.799909 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.799873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:03.923314 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.923247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:03.976376 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:03.976294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:04.103746 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.103597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:04.173301 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.173263 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78564f85d4-pxmtk"] Jul 06 23:44:04.178404 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.178364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.183511 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.183482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 23:44:04.183669 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.183567 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 23:44:04.183848 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.183829 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 23:44:04.184087 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.184071 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 23:44:04.184267 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.184251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 23:44:04.184347 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.183830 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vjf5d\"" Jul 06 23:44:04.185432 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.185377 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78564f85d4-pxmtk"] Jul 06 23:44:04.191733 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.191588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:04.283593 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.283554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-oauth-serving-cert\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.283775 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.283611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-oauth-config\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.283775 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.283660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-serving-cert\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.283775 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.283746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m59bg\" (UniqueName: \"kubernetes.io/projected/0ecf80be-4877-439b-adee-1cf4a635da0a-kube-api-access-m59bg\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.283931 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.283799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-console-config\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.283931 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.283856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-service-ca\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.385059 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.384828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-oauth-serving-cert\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.385219 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.385193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-oauth-config\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.385306 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.385236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-serving-cert\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.385376 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.385350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m59bg\" (UniqueName: \"kubernetes.io/projected/0ecf80be-4877-439b-adee-1cf4a635da0a-kube-api-access-m59bg\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.385433 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.385413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-console-config\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.385694 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.385477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-service-ca\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.385694 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.385681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-oauth-serving-cert\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.386166 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.386137 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-service-ca\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.386370 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.386266 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-console-config\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.389291 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.389133 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-serving-cert\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.389530 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.389511 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-oauth-config\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.394362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.394321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m59bg\" (UniqueName: \"kubernetes.io/projected/0ecf80be-4877-439b-adee-1cf4a635da0a-kube-api-access-m59bg\") pod \"console-78564f85d4-pxmtk\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:04.495381 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:04.495343 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:07.062357 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:07.062312 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:07.142844 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:07.142809 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:07.320250 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:07.320171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:07.423178 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:07.423134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:07.514902 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:07.514867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:08.318923 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:08.318872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:08.386463 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:08.386426 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:08.855010 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:08.854951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:08.955568 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:08.955533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:09.688416 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:09.688336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78564f85d4-pxmtk"] Jul 06 23:44:09.692423 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:09.692222 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecf80be_4877_439b_adee_1cf4a635da0a.slice/crio-20675f517ff0aafddf3f35adc7f72ad6e8b045820d9f8336b5a59cf8dbd3f70d WatchSource:0}: Error finding container 20675f517ff0aafddf3f35adc7f72ad6e8b045820d9f8336b5a59cf8dbd3f70d: Status 404 returned error can't find the container with id 20675f517ff0aafddf3f35adc7f72ad6e8b045820d9f8336b5a59cf8dbd3f70d Jul 06 23:44:10.037605 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.037539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:10.141999 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.141662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:10.202156 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.201519 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wpbdz" Jul 06 23:44:10.245752 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.245678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:10.304301 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.304266 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78564f85d4-pxmtk" event={"ID":"0ecf80be-4877-439b-adee-1cf4a635da0a","Type":"ContainerStarted","Data":"20675f517ff0aafddf3f35adc7f72ad6e8b045820d9f8336b5a59cf8dbd3f70d"} Jul 06 23:44:10.306384 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.306352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" event={"ID":"870abde4-501e-4a82-837d-abbb43a831cd","Type":"ContainerStarted","Data":"4cf9cd27f917c8f3f78554db51ed72edb76d3b6c0eae560a9a1d3d0dad695832"} Jul 06 23:44:10.309117 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.309076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" event={"ID":"dc6a3492-579a-4059-8ca1-7023d0b1a28a","Type":"ContainerStarted","Data":"f978d7c2d542674e97a5cf8e4e1604e86e2dabbcb21883cc095d41337f871348"} Jul 06 23:44:10.310200 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.309606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" event={"ID":"dc6a3492-579a-4059-8ca1-7023d0b1a28a","Type":"ContainerStarted","Data":"12b3e8b15a72a3a1ce15622a61c03360aeff92aee26a75343fca16482c422ea9"} Jul 06 23:44:10.312074 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.312048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" event={"ID":"586fd041-ba8d-4941-b9c5-39c9e2ae4d59","Type":"ContainerStarted","Data":"a59e899722cacb6d9db606fa9fda067747fa25684285914729ebafd808cf6a1b"} Jul 06 23:44:10.312180 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.312089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" event={"ID":"586fd041-ba8d-4941-b9c5-39c9e2ae4d59","Type":"ContainerStarted","Data":"527af582475341e61d8f3fd82f6c3110c75b93e673a7c841d86411c300b8bf7f"} Jul 06 23:44:10.312180 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.312121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" event={"ID":"586fd041-ba8d-4941-b9c5-39c9e2ae4d59","Type":"ContainerStarted","Data":"eb58e40f3e55cbfa19f6ac4d91bc0e7db8f47fa145b7b0842b0b2d4a33222750"} Jul 06 23:44:10.313754 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.313730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" event={"ID":"7b775ca7-c969-4e1c-a836-bcdaa9803f68","Type":"ContainerStarted","Data":"b23703153183b9c47de309470dbea6e875d6097b77207f602f5b06089c9f6123"} Jul 06 23:44:10.315836 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.315813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" event={"ID":"e0af8243-7525-4bca-9796-3f51c03a39e5","Type":"ContainerStarted","Data":"ddc935c1a6db49d72b3dd2e624a5837ae535ce99f9e484e126da272cee1ab3f0"} Jul 06 23:44:10.325753 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.325701 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-84wcc" podStartSLOduration=35.439038942 podStartE2EDuration="43.325681806s" totalImagesPullingTime="7.886642864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:10.324500593 +0000 UTC m=+80.351405826" watchObservedRunningTime="2026-07-06 23:44:10.325681806 +0000 UTC m=+80.352587036" Jul 06 23:44:10.362989 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.362927 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-675cb94cb4-6jqlt" podStartSLOduration=2.116366252 podStartE2EDuration="9.362905834s" totalImagesPullingTime="7.246539582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:10.361318993 +0000 UTC m=+80.388224225" watchObservedRunningTime="2026-07-06 23:44:10.362905834 +0000 UTC m=+80.389811066" Jul 06 23:44:10.363862 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.363469 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-697b8dbc9c-g9bpt" podStartSLOduration=2.081608097 podStartE2EDuration="9.363456511s" totalImagesPullingTime="7.281848414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:10.345057646 +0000 UTC m=+80.371962875" watchObservedRunningTime="2026-07-06 23:44:10.363456511 +0000 UTC m=+80.390361741" Jul 06 23:44:10.374833 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.374802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:10.384513 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.383822 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6rmbv" podStartSLOduration=2.515052084 podStartE2EDuration="9.383800491s" totalImagesPullingTime="6.868748407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:10.382137886 +0000 UTC m=+80.409043115" watchObservedRunningTime="2026-07-06 23:44:10.383800491 +0000 UTC m=+80.410705721" Jul 06 23:44:10.404126 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.404065 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-f227n" podStartSLOduration=34.788633683 podStartE2EDuration="42.404042317s" totalImagesPullingTime="7.615408634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:10.401818868 +0000 UTC m=+80.428724122" watchObservedRunningTime="2026-07-06 23:44:10.404042317 +0000 UTC m=+80.430947548" Jul 06 23:44:10.559225 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.559139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:10.620292 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:10.620243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:17.377354 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:17.377130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:17.475945 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:17.475907 2564 patch_prober.go:28] interesting pod/image-registry-76ccc5f9cc-m6kjg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 23:44:17.476113 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:17.475968 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" podUID="1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 23:44:18.216771 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:18.216730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:18.317136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:18.317101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:19.192353 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:19.192321 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:44:19.193037 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:19.193013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:19.279460 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:19.279418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:19.352769 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:19.352729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:19.454058 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:19.453967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:19.697438 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:19.697396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:21.289369 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:21.289336 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78564f85d4-pxmtk"] Jul 06 23:44:21.404415 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:21.404378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:21.505804 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:21.505768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:21.606076 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:21.605985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:21.669385 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:21.669351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:22.137967 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.137918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:22.237765 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.237741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:22.378137 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.377821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" event={"ID":"52c92b8b-0678-4c12-8737-4162ebb2f65e","Type":"ContainerStarted","Data":"b696e8e84f5d1187559d273667ad6b7142b1f6c32ae8e567fe109b95e41bb676"} Jul 06 23:44:22.378137 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.377886 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:22.379765 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.379741 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" Jul 06 23:44:22.379896 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.379873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78564f85d4-pxmtk" event={"ID":"0ecf80be-4877-439b-adee-1cf4a635da0a","Type":"ContainerStarted","Data":"0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305"} Jul 06 23:44:22.381841 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.381815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-bjj78" event={"ID":"bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1","Type":"ContainerStarted","Data":"e6efcb8df38f1560cf6a8d2aeb0c23135d04d57764be3ebcc4e4c9a4394ca75d"} Jul 06 23:44:22.382038 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.382019 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-bjj78" Jul 06 23:44:22.383044 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.383010 2564 patch_prober.go:28] interesting pod/downloads-685bfb5f64-bjj78 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.24:8080/\": dial tcp 10.132.0.24:8080: connect: connection refused" start-of-body= Jul 06 23:44:22.383156 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.383059 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-bjj78" podUID="bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.24:8080/\": dial tcp 10.132.0.24:8080: connect: connection refused" Jul 06 23:44:22.396659 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.395792 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b67bb96d7-s4qgm" podStartSLOduration=1.494791758 podStartE2EDuration="21.39577198s" totalImagesPullingTime="19.900980222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:22.394976184 +0000 UTC m=+92.421881416" watchObservedRunningTime="2026-07-06 23:44:22.39577198 +0000 UTC m=+92.422677209" Jul 06 23:44:22.434160 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.434109 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78564f85d4-pxmtk" podStartSLOduration=5.938653804 podStartE2EDuration="18.434093399s" totalImagesPullingTime="12.495439595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:22.432694736 +0000 UTC m=+92.459599966" watchObservedRunningTime="2026-07-06 23:44:22.434093399 +0000 UTC m=+92.460998628" Jul 06 23:44:22.451542 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:22.451483 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-bjj78" podStartSLOduration=1.5540969439999999 podStartE2EDuration="21.451463357s" totalImagesPullingTime="19.897366413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:22.449816271 +0000 UTC m=+92.476721500" watchObservedRunningTime="2026-07-06 23:44:22.451463357 +0000 UTC m=+92.478368635" Jul 06 23:44:23.365409 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.365373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:23.403233 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.403200 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-bjj78" Jul 06 23:44:23.594365 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.594330 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:23.670247 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.670160 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qx4zk"] Jul 06 23:44:23.691704 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.691665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:23.702179 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.702142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.714315 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.714277 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-57rgw\"" Jul 06 23:44:23.714586 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.714568 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 23:44:23.714810 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.714794 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 23:44:23.715076 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.715059 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 23:44:23.715306 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.715291 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 23:44:23.762376 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-root\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-tls\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762415 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-textfile\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-wtmp\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-sys\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762581 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9r6f\" (UniqueName: \"kubernetes.io/projected/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-kube-api-access-n9r6f\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762916 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.762916 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.762620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-metrics-client-ca\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.826521 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.826483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-root\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-tls\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-textfile\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-wtmp\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-sys\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9r6f\" (UniqueName: \"kubernetes.io/projected/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-kube-api-access-n9r6f\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.867648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.867380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-metrics-client-ca\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.868203 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.868153 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-metrics-client-ca\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.871642 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.868322 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-wtmp\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.871642 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.868374 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-root\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.871642 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.869377 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-textfile\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.871642 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.870193 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.871642 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.870261 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-sys\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.887117 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.880591 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.887117 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.885621 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9r6f\" (UniqueName: \"kubernetes.io/projected/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-kube-api-access-n9r6f\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.888455 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.888414 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0-node-exporter-tls\") pod \"node-exporter-qx4zk\" (UID: \"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0\") " pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:23.931763 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.931589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:23.989600 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:23.989514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:24.030800 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.030513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qx4zk" Jul 06 23:44:24.075453 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:44:24.075417 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78d8a51_0d3d_4b8b_a93e_e2b41787f4c0.slice/crio-02d49e628643482131be97383b96aa4cf8781ab9fb6b5a3fc559c738fc9a5bbc WatchSource:0}: Error finding container 02d49e628643482131be97383b96aa4cf8781ab9fb6b5a3fc559c738fc9a5bbc: Status 404 returned error can't find the container with id 02d49e628643482131be97383b96aa4cf8781ab9fb6b5a3fc559c738fc9a5bbc Jul 06 23:44:24.197771 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.197685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:24.394228 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.393885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qx4zk" event={"ID":"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0","Type":"ContainerStarted","Data":"02d49e628643482131be97383b96aa4cf8781ab9fb6b5a3fc559c738fc9a5bbc"} Jul 06 23:44:24.465517 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.465434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:24.496220 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.496184 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:24.581257 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.577907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:24.657285 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.657251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:24.882702 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.882652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:24.984811 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:24.984774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:26.405453 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:26.405415 2564 generic.go:356] "Generic (PLEG): container finished" podID="a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0" containerID="1536e81080ff9f4f7d5aad13494350284220f892d5ac3fa5dbba74eeaf29837f" exitCode=0 Jul 06 23:44:26.405910 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:26.405502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qx4zk" event={"ID":"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0","Type":"ContainerDied","Data":"1536e81080ff9f4f7d5aad13494350284220f892d5ac3fa5dbba74eeaf29837f"} Jul 06 23:44:26.541043 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:26.541003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:26.641154 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:26.641085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:26.747406 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:26.747373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:26.850737 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:26.850699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:27.413462 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:27.413429 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qx4zk" event={"ID":"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0","Type":"ContainerStarted","Data":"2cb84ebd3c70be59fae0ecb39d4c95fba8149185683c925f74b3f39c328113e2"} Jul 06 23:44:27.413942 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:27.413485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qx4zk" event={"ID":"a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0","Type":"ContainerStarted","Data":"6e18141ad274b0787950a26715cc93925d90063be5cb3be77dac6f503c90e87c"} Jul 06 23:44:27.434961 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:27.434912 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qx4zk" podStartSLOduration=2.8392407349999997 podStartE2EDuration="4.434892551s" totalImagesPullingTime="1.483953924s" totalInitContainerRuntime="111.697892ms" isStatefulPod=true podCreationTimestamp="2026-07-06 23:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:27.433465224 +0000 UTC m=+97.460370454" watchObservedRunningTime="2026-07-06 23:44:27.434892551 +0000 UTC m=+97.461797785" Jul 06 23:44:28.405369 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:28.405329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:28.494972 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:28.494932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:28.595586 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:28.595544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:28.695207 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:28.695128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:29.092256 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:29.092220 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76ccc5f9cc-m6kjg"] Jul 06 23:44:29.177753 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:29.177714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:29.190209 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:29.190154 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jj75g" Jul 06 23:44:29.278501 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:29.278466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:29.831292 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:29.831255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:30.369979 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:30.369940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:30.470352 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:30.470307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:30.526444 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:30.526405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:32.367799 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:32.367764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:32.420846 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:32.420811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:33.050464 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:33.050430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:33.151295 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:33.151259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:33.212413 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:33.212379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:33.926179 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:33.926134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:33.978397 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:33.978360 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:36.470643 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:36.470599 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:36.572517 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:36.572484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:39.499465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:39.499433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:39.601373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:39.601334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:39.700113 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:39.700075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:40.719306 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:40.719273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:40.819774 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:40.819745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:40.870492 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:40.870459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:41.358305 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:41.358273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:41.413273 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:41.413245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:42.125277 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:42.125241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:42.227056 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:42.226494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:44.486463 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:44.486383 2564 generic.go:356] "Generic (PLEG): container finished" podID="3514de1e-a4bb-4da8-b18c-a8f12dd150bd" containerID="8e24e872bab4e0f11e0446eabfd6192b383dc5f8024484dad6dc9eb57c88b796" exitCode=0 Jul 06 23:44:44.486830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:44.486464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" event={"ID":"3514de1e-a4bb-4da8-b18c-a8f12dd150bd","Type":"ContainerDied","Data":"8e24e872bab4e0f11e0446eabfd6192b383dc5f8024484dad6dc9eb57c88b796"} Jul 06 23:44:44.486830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:44.486817 2564 scope.go:117] "RemoveContainer" containerID="8e24e872bab4e0f11e0446eabfd6192b383dc5f8024484dad6dc9eb57c88b796" Jul 06 23:44:45.492327 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:45.492296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-cdx5w" event={"ID":"3514de1e-a4bb-4da8-b18c-a8f12dd150bd","Type":"ContainerStarted","Data":"7199e4f676f0a126f5787d6e57726aa4e9e79c550c82fe1951dded79cad904f5"} Jul 06 23:44:47.406213 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.406171 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-78564f85d4-pxmtk" podUID="0ecf80be-4877-439b-adee-1cf4a635da0a" containerName="console" containerID="cri-o://0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305" gracePeriod=15 Jul 06 23:44:47.675448 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.675374 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:47.775145 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775117 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m59bg\" (UniqueName: \"kubernetes.io/projected/0ecf80be-4877-439b-adee-1cf4a635da0a-kube-api-access-m59bg\") pod \"0ecf80be-4877-439b-adee-1cf4a635da0a\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " Jul 06 23:44:47.775306 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775152 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-oauth-serving-cert\") pod \"0ecf80be-4877-439b-adee-1cf4a635da0a\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " Jul 06 23:44:47.775306 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775180 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-oauth-config\") pod \"0ecf80be-4877-439b-adee-1cf4a635da0a\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " Jul 06 23:44:47.775306 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775202 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-serving-cert\") pod \"0ecf80be-4877-439b-adee-1cf4a635da0a\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " Jul 06 23:44:47.775442 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775374 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-console-config\") pod \"0ecf80be-4877-439b-adee-1cf4a635da0a\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " Jul 06 23:44:47.775442 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775433 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-service-ca\") pod \"0ecf80be-4877-439b-adee-1cf4a635da0a\" (UID: \"0ecf80be-4877-439b-adee-1cf4a635da0a\") " Jul 06 23:44:47.775559 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775526 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0ecf80be-4877-439b-adee-1cf4a635da0a" (UID: "0ecf80be-4877-439b-adee-1cf4a635da0a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:44:47.776157 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.775735 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-console-config" (OuterVolumeSpecName: "console-config") pod "0ecf80be-4877-439b-adee-1cf4a635da0a" (UID: "0ecf80be-4877-439b-adee-1cf4a635da0a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:44:47.776157 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.776133 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-service-ca" (OuterVolumeSpecName: "service-ca") pod "0ecf80be-4877-439b-adee-1cf4a635da0a" (UID: "0ecf80be-4877-439b-adee-1cf4a635da0a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:44:47.776370 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.776346 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-oauth-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:47.781674 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.778351 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecf80be-4877-439b-adee-1cf4a635da0a-kube-api-access-m59bg" (OuterVolumeSpecName: "kube-api-access-m59bg") pod "0ecf80be-4877-439b-adee-1cf4a635da0a" (UID: "0ecf80be-4877-439b-adee-1cf4a635da0a"). InnerVolumeSpecName "kube-api-access-m59bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:44:47.781786 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.781741 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0ecf80be-4877-439b-adee-1cf4a635da0a" (UID: "0ecf80be-4877-439b-adee-1cf4a635da0a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:44:47.782392 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.782368 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0ecf80be-4877-439b-adee-1cf4a635da0a" (UID: "0ecf80be-4877-439b-adee-1cf4a635da0a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:44:47.877291 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.877255 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-oauth-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:47.877291 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.877285 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ecf80be-4877-439b-adee-1cf4a635da0a-console-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:47.877291 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.877295 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-console-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:47.877523 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.877305 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecf80be-4877-439b-adee-1cf4a635da0a-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:47.877523 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:47.877336 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m59bg\" (UniqueName: \"kubernetes.io/projected/0ecf80be-4877-439b-adee-1cf4a635da0a-kube-api-access-m59bg\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:48.504888 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.504848 2564 generic.go:356] "Generic (PLEG): container finished" podID="0ecf80be-4877-439b-adee-1cf4a635da0a" containerID="0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305" exitCode=0 Jul 06 23:44:48.505429 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.504925 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78564f85d4-pxmtk" Jul 06 23:44:48.505429 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.504934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78564f85d4-pxmtk" event={"ID":"0ecf80be-4877-439b-adee-1cf4a635da0a","Type":"ContainerDied","Data":"0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305"} Jul 06 23:44:48.505429 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.505012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78564f85d4-pxmtk" event={"ID":"0ecf80be-4877-439b-adee-1cf4a635da0a","Type":"ContainerDied","Data":"20675f517ff0aafddf3f35adc7f72ad6e8b045820d9f8336b5a59cf8dbd3f70d"} Jul 06 23:44:48.505429 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.505043 2564 scope.go:117] "RemoveContainer" containerID="0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305" Jul 06 23:44:48.514541 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.514522 2564 scope.go:117] "RemoveContainer" containerID="0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305" Jul 06 23:44:48.514817 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:44:48.514795 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305\": container with ID starting with 0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305 not found: ID does not exist" containerID="0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305" Jul 06 23:44:48.514892 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.514829 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305"} err="failed to get container status \"0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305\": rpc error: code = NotFound desc = could not find container \"0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305\": container with ID starting with 0b23335ec5e33bebd426b6e78edee2ea71a1b6090bb6fc8f932037d84be36305 not found: ID does not exist" Jul 06 23:44:48.525214 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.525193 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78564f85d4-pxmtk"] Jul 06 23:44:48.528604 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.528585 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78564f85d4-pxmtk"] Jul 06 23:44:48.654112 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.654076 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ecf80be-4877-439b-adee-1cf4a635da0a" path="/var/lib/kubelet/pods/0ecf80be-4877-439b-adee-1cf4a635da0a/volumes" Jul 06 23:44:48.919449 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.919419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:48.972591 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:48.972556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:44:50.514049 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:50.513979 2564 generic.go:356] "Generic (PLEG): container finished" podID="539120a9-c4eb-424e-942b-f7f2047712b7" containerID="fa34d1d679df86007b36ec6148da99afeda939cb2cd926aa60618bd96d46920f" exitCode=0 Jul 06 23:44:50.514366 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:50.514052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" event={"ID":"539120a9-c4eb-424e-942b-f7f2047712b7","Type":"ContainerDied","Data":"fa34d1d679df86007b36ec6148da99afeda939cb2cd926aa60618bd96d46920f"} Jul 06 23:44:50.514366 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:50.514354 2564 scope.go:117] "RemoveContainer" containerID="fa34d1d679df86007b36ec6148da99afeda939cb2cd926aa60618bd96d46920f" Jul 06 23:44:51.524040 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:51.524007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b97bk" event={"ID":"539120a9-c4eb-424e-942b-f7f2047712b7","Type":"ContainerStarted","Data":"1e8dcf810b2bafbc6a761193841598d6779de2ae88d5eb9712dd16681b7019e1"} Jul 06 23:44:54.123842 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.123804 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" podUID="1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" containerName="registry" containerID="cri-o://4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6" gracePeriod=30 Jul 06 23:44:54.403061 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.403003 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:44:54.431382 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431357 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-certificates\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431536 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431414 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-trusted-ca\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431597 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431551 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431621 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-bound-sa-token\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431725 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431689 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-ca-trust-extracted\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431725 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431718 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-installation-pull-secrets\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431833 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431755 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-image-registry-private-configuration\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431833 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431791 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58n4h\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-kube-api-access-58n4h\") pod \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\" (UID: \"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e\") " Jul 06 23:44:54.431920 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431826 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:44:54.431920 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.431841 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:44:54.432032 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.432012 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-certificates\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.432080 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.432033 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-trusted-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.434415 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.434386 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:44:54.434555 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.434510 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:44:54.434646 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.434607 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:44:54.434791 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.434772 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-kube-api-access-58n4h" (OuterVolumeSpecName: "kube-api-access-58n4h") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "kube-api-access-58n4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:44:54.434843 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.434790 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:44:54.443224 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.443199 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" (UID: "1e1fa4c9-5f42-4ee6-89ab-7f735b43977e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:44:54.533286 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.533240 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-bound-sa-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.533286 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.533276 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-ca-trust-extracted\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.533286 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.533286 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-installation-pull-secrets\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.533286 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.533295 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-image-registry-private-configuration\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.533659 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.533305 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58n4h\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-kube-api-access-58n4h\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.533659 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.533315 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e-registry-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:44:54.537768 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.537734 2564 generic.go:356] "Generic (PLEG): container finished" podID="1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" containerID="4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6" exitCode=0 Jul 06 23:44:54.537917 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.537785 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" Jul 06 23:44:54.537917 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.537821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" event={"ID":"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e","Type":"ContainerDied","Data":"4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6"} Jul 06 23:44:54.537917 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.537874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ccc5f9cc-m6kjg" event={"ID":"1e1fa4c9-5f42-4ee6-89ab-7f735b43977e","Type":"ContainerDied","Data":"431a665e305f670321106421c1523db8bdb66c42132e48158386f5c41b3b52ce"} Jul 06 23:44:54.537917 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.537894 2564 scope.go:117] "RemoveContainer" containerID="4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6" Jul 06 23:44:54.546564 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.546548 2564 scope.go:117] "RemoveContainer" containerID="4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6" Jul 06 23:44:54.546874 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:44:54.546853 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6\": container with ID starting with 4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6 not found: ID does not exist" containerID="4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6" Jul 06 23:44:54.546940 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.546883 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6"} err="failed to get container status \"4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6\": rpc error: code = NotFound desc = could not find container \"4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6\": container with ID starting with 4ce53faa0cac20077b651932c2f15c5425023e079ff254672c721bb0d62b09c6 not found: ID does not exist" Jul 06 23:44:54.557931 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.557902 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76ccc5f9cc-m6kjg"] Jul 06 23:44:54.561542 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.561517 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76ccc5f9cc-m6kjg"] Jul 06 23:44:54.654474 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:54.654398 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" path="/var/lib/kubelet/pods/1e1fa4c9-5f42-4ee6-89ab-7f735b43977e/volumes" Jul 06 23:44:58.555558 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:58.555524 2564 generic.go:356] "Generic (PLEG): container finished" podID="438b86b7-b5b2-4243-a133-af4c044f9b57" containerID="a278a79e8c69016cd379e78fbec33c58655e95c899aca224a3962aa24f2cb9a1" exitCode=0 Jul 06 23:44:58.556027 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:58.555606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" event={"ID":"438b86b7-b5b2-4243-a133-af4c044f9b57","Type":"ContainerDied","Data":"a278a79e8c69016cd379e78fbec33c58655e95c899aca224a3962aa24f2cb9a1"} Jul 06 23:44:58.556082 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:58.556068 2564 scope.go:117] "RemoveContainer" containerID="a278a79e8c69016cd379e78fbec33c58655e95c899aca224a3962aa24f2cb9a1" Jul 06 23:44:59.560960 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:44:59.560924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jzx9r" event={"ID":"438b86b7-b5b2-4243-a133-af4c044f9b57","Type":"ContainerStarted","Data":"b6cc8b5dd470c37be4f23ac052037bb0c262fc14919389811e7efabea38d7667"} Jul 06 23:45:03.925574 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:03.925536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:03.973261 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:03.973221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:19.878473 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:19.878439 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:19.933297 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:19.933254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:42.799082 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:42.798960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:43.052464 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:43.052378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:52.394102 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:52.394066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:52.447590 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:52.447552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:58.010756 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:58.010714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:45:58.064685 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:45:58.064622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:02.767760 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:02.767721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:02.828081 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:02.828032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:03.375712 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:03.375677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:03.426362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:03.426332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:09.962531 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:09.962494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:10.015742 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:10.015712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:13.863424 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:13.863386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:13.926641 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:13.926596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:13.977360 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:13.977327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:18.952534 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:18.952456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:19.040152 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:19.040106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:19.094994 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:19.094966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:24.247894 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:24.247858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:28.124997 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:28.124956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:29.372994 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:29.372954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:29.474590 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:29.474543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:46:43.775617 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:46:43.775581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:08.394454 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:08.394415 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:08.610294 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:08.610256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:30.554895 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:30.554812 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:31.338348 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:31.338304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:34.167225 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:34.167173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:34.266799 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:34.266759 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:34.860069 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:34.860035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:34.965887 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:34.965856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:35.497367 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:35.497330 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:35.597693 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:35.597654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:37.265254 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:37.265222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:37.470461 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:37.470428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:40.753014 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:40.752971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:40.853490 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:40.853452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:44.543615 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:44.543577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:48.765896 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:48.765861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:48.867109 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:48.867073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:50.393338 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:50.393307 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:47:50.397134 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:50.397111 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:47:50.398373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:50.398346 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:47:50.401906 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:50.401888 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:47:50.407150 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:50.407118 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 23:47:53.884548 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:53.884511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:54.926668 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:54.926617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:47:55.968199 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:47:55.968166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:09.660396 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:09.660349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:09.761903 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:09.761868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:12.839639 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:12.839607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:12.954234 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:12.954200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:13.391620 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.391585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:13.601712 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.601660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:13.892373 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.892340 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr"] Jul 06 23:48:13.892777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.892679 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" containerName="registry" Jul 06 23:48:13.892777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.892692 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" containerName="registry" Jul 06 23:48:13.892777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.892701 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecf80be-4877-439b-adee-1cf4a635da0a" containerName="console" Jul 06 23:48:13.892777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.892706 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecf80be-4877-439b-adee-1cf4a635da0a" containerName="console" Jul 06 23:48:13.892777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.892769 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e1fa4c9-5f42-4ee6-89ab-7f735b43977e" containerName="registry" Jul 06 23:48:13.892777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.892778 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ecf80be-4877-439b-adee-1cf4a635da0a" containerName="console" Jul 06 23:48:13.894611 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.894595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:13.896960 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.896937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 23:48:13.897201 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.897163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 23:48:13.897317 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.897279 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 23:48:13.897385 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.897320 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 23:48:13.897385 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.897337 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v8hd2\"" Jul 06 23:48:13.915508 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.915471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr"] Jul 06 23:48:13.961282 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.961248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:13.992724 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.992690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:13.992863 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.992736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-tmp\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:13.992863 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.992766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6t8\" (UniqueName: \"kubernetes.io/projected/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-kube-api-access-jr6t8\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:13.992863 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.992827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-manifests\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:13.992863 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:13.992852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.063824 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.063789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:14.093904 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.093865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6t8\" (UniqueName: \"kubernetes.io/projected/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-kube-api-access-jr6t8\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.093904 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.093905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-manifests\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.094141 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.093926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.094141 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.093982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.094141 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.094015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-tmp\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.094314 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.094286 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-manifests\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.096255 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.096228 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-tmp\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.096579 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.096558 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.096677 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.096575 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.107026 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.106998 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6t8\" (UniqueName: \"kubernetes.io/projected/a1510dbb-cae6-4353-8bad-e7ddda73a8ac-kube-api-access-jr6t8\") pod \"opendatahub-operator-controller-manager-6688846b76-hwrjr\" (UID: \"a1510dbb-cae6-4353-8bad-e7ddda73a8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.166417 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.166326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:14.204541 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.204510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:14.278658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.278599 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:14.368837 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.368811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr"] Jul 06 23:48:14.371656 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:48:14.371599 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1510dbb_cae6_4353_8bad_e7ddda73a8ac.slice/crio-ee7c6034ecd18b583fcc76618d8d94a73c8fbde5823efd80e79f807d93cb3392 WatchSource:0}: Error finding container ee7c6034ecd18b583fcc76618d8d94a73c8fbde5823efd80e79f807d93cb3392: Status 404 returned error can't find the container with id ee7c6034ecd18b583fcc76618d8d94a73c8fbde5823efd80e79f807d93cb3392 Jul 06 23:48:14.373424 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.373408 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:48:14.413057 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.413022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:14.513919 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.513838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:14.613890 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:14.613846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:15.334447 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:15.334409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" event={"ID":"a1510dbb-cae6-4353-8bad-e7ddda73a8ac","Type":"ContainerStarted","Data":"ee7c6034ecd18b583fcc76618d8d94a73c8fbde5823efd80e79f807d93cb3392"} Jul 06 23:48:17.345760 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:17.345722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" event={"ID":"a1510dbb-cae6-4353-8bad-e7ddda73a8ac","Type":"ContainerStarted","Data":"b427553b57527665f6a96a8baed0344ffe25efced79a0c03c71c5318a60f647f"} Jul 06 23:48:18.351061 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:18.351026 2564 generic.go:356] "Generic (PLEG): container finished" podID="a1510dbb-cae6-4353-8bad-e7ddda73a8ac" containerID="b427553b57527665f6a96a8baed0344ffe25efced79a0c03c71c5318a60f647f" exitCode=0 Jul 06 23:48:18.351456 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:18.351092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" event={"ID":"a1510dbb-cae6-4353-8bad-e7ddda73a8ac","Type":"ContainerDied","Data":"b427553b57527665f6a96a8baed0344ffe25efced79a0c03c71c5318a60f647f"} Jul 06 23:48:18.376294 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:18.372384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:18.491050 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:18.491009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:19.358282 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:19.358247 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" event={"ID":"a1510dbb-cae6-4353-8bad-e7ddda73a8ac","Type":"ContainerStarted","Data":"0452f9ea1f01a3cde411ac923a76d94220f7af4268c45c83b047315dc3280235"} Jul 06 23:48:19.358775 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:19.358336 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:19.415155 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:19.415089 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" podStartSLOduration=2.021422418 podStartE2EDuration="6.415067229s" totalImagesPullingTime="3.251979686s" totalInitContainerRuntime="1.141665125s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:48:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:48:19.412723839 +0000 UTC m=+329.439629068" watchObservedRunningTime="2026-07-06 23:48:19.415067229 +0000 UTC m=+329.441972458" Jul 06 23:48:25.358428 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:25.358372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:25.455760 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:25.455722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:30.364549 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:30.364523 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6688846b76-hwrjr" Jul 06 23:48:34.499342 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:34.499303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:34.659996 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:34.659953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:34.930435 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:34.930388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:35.137275 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:35.137236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:35.544959 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:35.544924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:35.645585 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:35.645543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:35.745494 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:35.745447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:35.860748 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:35.860647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:36.474249 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:36.474213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:36.552126 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:36.552093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:36.632584 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:36.632541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:37.571458 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:37.571412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:40.521716 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:40.521685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:40.583190 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:40.583155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:41.097720 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:41.097688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:41.314702 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:41.314663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:41.451282 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:41.451186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:42.909689 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.909601 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg"] Jul 06 23:48:42.924762 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.924724 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg"] Jul 06 23:48:42.924923 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.924817 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:42.927095 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.927076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 23:48:42.927908 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.927883 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 23:48:42.927908 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.927903 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:48:42.928070 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.927931 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 23:48:42.928155 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.928136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 23:48:42.928282 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:42.928265 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qczgf\"" Jul 06 23:48:43.041077 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.041040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxh9\" (UniqueName: \"kubernetes.io/projected/3a39751b-8342-4db8-af20-99546112cc84-kube-api-access-tmxh9\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.041240 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.041084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a39751b-8342-4db8-af20-99546112cc84-cert\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.041240 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.041108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a39751b-8342-4db8-af20-99546112cc84-metrics-cert\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.041331 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.041271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a39751b-8342-4db8-af20-99546112cc84-manager-config\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.142219 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.142185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a39751b-8342-4db8-af20-99546112cc84-manager-config\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.142398 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.142234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxh9\" (UniqueName: \"kubernetes.io/projected/3a39751b-8342-4db8-af20-99546112cc84-kube-api-access-tmxh9\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.142398 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.142337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a39751b-8342-4db8-af20-99546112cc84-cert\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.142398 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.142373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a39751b-8342-4db8-af20-99546112cc84-metrics-cert\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.142835 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.142817 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a39751b-8342-4db8-af20-99546112cc84-manager-config\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.144923 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.144900 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a39751b-8342-4db8-af20-99546112cc84-metrics-cert\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.145017 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.144900 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a39751b-8342-4db8-af20-99546112cc84-cert\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.151591 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.151568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxh9\" (UniqueName: \"kubernetes.io/projected/3a39751b-8342-4db8-af20-99546112cc84-kube-api-access-tmxh9\") pod \"lws-controller-manager-6557895b7f-2czcg\" (UID: \"3a39751b-8342-4db8-af20-99546112cc84\") " pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.235272 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.235161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:48:43.380101 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.380067 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg"] Jul 06 23:48:43.383390 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:48:43.383362 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a39751b_8342_4db8_af20_99546112cc84.slice/crio-03f4276cf5b0e9087c10f8c153e43c54a45133e484d0aa6cb6614dac4ce440c0 WatchSource:0}: Error finding container 03f4276cf5b0e9087c10f8c153e43c54a45133e484d0aa6cb6614dac4ce440c0: Status 404 returned error can't find the container with id 03f4276cf5b0e9087c10f8c153e43c54a45133e484d0aa6cb6614dac4ce440c0 Jul 06 23:48:43.467818 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:43.467784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" event={"ID":"3a39751b-8342-4db8-af20-99546112cc84","Type":"ContainerStarted","Data":"03f4276cf5b0e9087c10f8c153e43c54a45133e484d0aa6cb6614dac4ce440c0"} Jul 06 23:48:45.489460 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:45.489415 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:45.596928 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:45.596893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:45.987676 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:45.987641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:46.075199 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:46.075167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:46.778163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:46.778132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:46.877371 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:46.877328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:47.009442 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:47.009413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:59.672777 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:59.672733 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:48:59.774282 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:48:59.774247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:49:01.519992 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:01.519960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:49:01.571869 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:01.571839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:49:05.567712 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:05.567673 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" event={"ID":"3a39751b-8342-4db8-af20-99546112cc84","Type":"ContainerStarted","Data":"447a8897e11c4358e9884386dfc1e991715d60b58388b082f898cef8f49316bc"} Jul 06 23:49:05.568085 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:05.567741 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:49:05.583485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:05.583443 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" podStartSLOduration=2.392204906 podStartE2EDuration="23.583429748s" totalImagesPullingTime="21.191224842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:05.582979572 +0000 UTC m=+375.609884801" watchObservedRunningTime="2026-07-06 23:49:05.583429748 +0000 UTC m=+375.610334977" Jul 06 23:49:16.574415 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:16.574381 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6557895b7f-2czcg" Jul 06 23:49:36.115353 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.115310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:49:36.215778 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.215742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:49:36.222711 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.222684 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tltrp"] Jul 06 23:49:36.229471 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.229453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" Jul 06 23:49:36.231980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.231964 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xc6fh\"" Jul 06 23:49:36.232715 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.232698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.232801 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.232709 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.240352 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.240328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tltrp"] Jul 06 23:49:36.294368 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.294332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5twqw\" (UniqueName: \"kubernetes.io/projected/916ca4b7-f10b-47c0-b576-4f2371a552a0-kube-api-access-5twqw\") pod \"kuadrant-operator-catalog-tltrp\" (UID: \"916ca4b7-f10b-47c0-b576-4f2371a552a0\") " pod="kuadrant-system/kuadrant-operator-catalog-tltrp" Jul 06 23:49:36.352183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.352148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:49:36.395336 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.395251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5twqw\" (UniqueName: \"kubernetes.io/projected/916ca4b7-f10b-47c0-b576-4f2371a552a0-kube-api-access-5twqw\") pod \"kuadrant-operator-catalog-tltrp\" (UID: \"916ca4b7-f10b-47c0-b576-4f2371a552a0\") " pod="kuadrant-system/kuadrant-operator-catalog-tltrp" Jul 06 23:49:36.405276 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.405249 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5twqw\" (UniqueName: \"kubernetes.io/projected/916ca4b7-f10b-47c0-b576-4f2371a552a0-kube-api-access-5twqw\") pod \"kuadrant-operator-catalog-tltrp\" (UID: \"916ca4b7-f10b-47c0-b576-4f2371a552a0\") " pod="kuadrant-system/kuadrant-operator-catalog-tltrp" Jul 06 23:49:36.540208 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.540167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" Jul 06 23:49:36.669418 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.669343 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tltrp"] Jul 06 23:49:36.671941 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:49:36.671913 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916ca4b7_f10b_47c0_b576_4f2371a552a0.slice/crio-ed18e8bc491894c03d06aac5292111f5020f98636bf3d56192d3ce8a0b5c2c54 WatchSource:0}: Error finding container ed18e8bc491894c03d06aac5292111f5020f98636bf3d56192d3ce8a0b5c2c54: Status 404 returned error can't find the container with id ed18e8bc491894c03d06aac5292111f5020f98636bf3d56192d3ce8a0b5c2c54 Jul 06 23:49:36.698929 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.698893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" event={"ID":"916ca4b7-f10b-47c0-b576-4f2371a552a0","Type":"ContainerStarted","Data":"ed18e8bc491894c03d06aac5292111f5020f98636bf3d56192d3ce8a0b5c2c54"} Jul 06 23:49:36.745554 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.745525 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tltrp"] Jul 06 23:49:36.802263 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.802233 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hdjb4"] Jul 06 23:49:36.806532 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.806513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:36.815671 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.815649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hdjb4"] Jul 06 23:49:36.900027 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:36.899992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdk9\" (UniqueName: \"kubernetes.io/projected/aa521a3e-2db8-4adb-bb07-ecba685763f3-kube-api-access-7pdk9\") pod \"kuadrant-operator-catalog-hdjb4\" (UID: \"aa521a3e-2db8-4adb-bb07-ecba685763f3\") " pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:37.001388 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:37.001295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdk9\" (UniqueName: \"kubernetes.io/projected/aa521a3e-2db8-4adb-bb07-ecba685763f3-kube-api-access-7pdk9\") pod \"kuadrant-operator-catalog-hdjb4\" (UID: \"aa521a3e-2db8-4adb-bb07-ecba685763f3\") " pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:37.009103 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:37.009078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdk9\" (UniqueName: \"kubernetes.io/projected/aa521a3e-2db8-4adb-bb07-ecba685763f3-kube-api-access-7pdk9\") pod \"kuadrant-operator-catalog-hdjb4\" (UID: \"aa521a3e-2db8-4adb-bb07-ecba685763f3\") " pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:37.117054 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:37.117020 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:37.275899 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:37.275842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hdjb4"] Jul 06 23:49:37.288415 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:49:37.288387 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa521a3e_2db8_4adb_bb07_ecba685763f3.slice/crio-3e0d5737b5d6c8133f9b6dd2ded62c78e22465be4e6b4030cce37f374b33e3be WatchSource:0}: Error finding container 3e0d5737b5d6c8133f9b6dd2ded62c78e22465be4e6b4030cce37f374b33e3be: Status 404 returned error can't find the container with id 3e0d5737b5d6c8133f9b6dd2ded62c78e22465be4e6b4030cce37f374b33e3be Jul 06 23:49:37.705795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:37.705679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" event={"ID":"aa521a3e-2db8-4adb-bb07-ecba685763f3","Type":"ContainerStarted","Data":"3e0d5737b5d6c8133f9b6dd2ded62c78e22465be4e6b4030cce37f374b33e3be"} Jul 06 23:49:38.712879 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:38.712830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" event={"ID":"916ca4b7-f10b-47c0-b576-4f2371a552a0","Type":"ContainerStarted","Data":"79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3"} Jul 06 23:49:38.713352 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:38.712983 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" podUID="916ca4b7-f10b-47c0-b576-4f2371a552a0" containerName="registry-server" containerID="cri-o://79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3" gracePeriod=2 Jul 06 23:49:38.714707 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:38.714679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" event={"ID":"aa521a3e-2db8-4adb-bb07-ecba685763f3","Type":"ContainerStarted","Data":"2ee7a37b270db531e8f62a1f5f02f53e04f14f066436db68785c4b8c301f8f85"} Jul 06 23:49:38.752037 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:38.751986 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" podStartSLOduration=0.823288916 podStartE2EDuration="2.751968521s" totalImagesPullingTime="1.928679605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:38.732932734 +0000 UTC m=+408.759837965" watchObservedRunningTime="2026-07-06 23:49:38.751968521 +0000 UTC m=+408.778873750" Jul 06 23:49:38.752194 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:38.752068 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" podStartSLOduration=1.439187715 podStartE2EDuration="2.752062946s" totalImagesPullingTime="1.312875231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:38.751854423 +0000 UTC m=+408.778759652" watchObservedRunningTime="2026-07-06 23:49:38.752062946 +0000 UTC m=+408.778968174" Jul 06 23:49:38.958302 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:38.958278 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tltrp_916ca4b7-f10b-47c0-b576-4f2371a552a0/registry-server/0.log" Jul 06 23:49:38.960649 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:38.960616 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" Jul 06 23:49:39.018135 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.018096 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5twqw\" (UniqueName: \"kubernetes.io/projected/916ca4b7-f10b-47c0-b576-4f2371a552a0-kube-api-access-5twqw\") pod \"916ca4b7-f10b-47c0-b576-4f2371a552a0\" (UID: \"916ca4b7-f10b-47c0-b576-4f2371a552a0\") " Jul 06 23:49:39.020317 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.020290 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916ca4b7-f10b-47c0-b576-4f2371a552a0-kube-api-access-5twqw" (OuterVolumeSpecName: "kube-api-access-5twqw") pod "916ca4b7-f10b-47c0-b576-4f2371a552a0" (UID: "916ca4b7-f10b-47c0-b576-4f2371a552a0"). InnerVolumeSpecName "kube-api-access-5twqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:49:39.119468 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.119430 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5twqw\" (UniqueName: \"kubernetes.io/projected/916ca4b7-f10b-47c0-b576-4f2371a552a0-kube-api-access-5twqw\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:49:39.720412 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.720383 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tltrp_916ca4b7-f10b-47c0-b576-4f2371a552a0/registry-server/0.log" Jul 06 23:49:39.720796 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.720424 2564 generic.go:356] "Generic (PLEG): container finished" podID="916ca4b7-f10b-47c0-b576-4f2371a552a0" containerID="79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3" exitCode=2 Jul 06 23:49:39.720796 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.720476 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" Jul 06 23:49:39.720796 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.720517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" event={"ID":"916ca4b7-f10b-47c0-b576-4f2371a552a0","Type":"ContainerDied","Data":"79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3"} Jul 06 23:49:39.720796 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.720568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tltrp" event={"ID":"916ca4b7-f10b-47c0-b576-4f2371a552a0","Type":"ContainerDied","Data":"ed18e8bc491894c03d06aac5292111f5020f98636bf3d56192d3ce8a0b5c2c54"} Jul 06 23:49:39.720796 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.720590 2564 scope.go:117] "RemoveContainer" containerID="79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3" Jul 06 23:49:39.729045 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.729023 2564 scope.go:117] "RemoveContainer" containerID="79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3" Jul 06 23:49:39.729313 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:49:39.729295 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3\": container with ID starting with 79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3 not found: ID does not exist" containerID="79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3" Jul 06 23:49:39.729366 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.729321 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3"} err="failed to get container status \"79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3\": rpc error: code = NotFound desc = could not find container \"79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3\": container with ID starting with 79dc218889c679011d2b564552871d9e2fba399b725d5360a3cd487f1996bae3 not found: ID does not exist" Jul 06 23:49:39.743595 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.743571 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tltrp"] Jul 06 23:49:39.750744 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:39.750722 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tltrp"] Jul 06 23:49:40.654376 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:40.654344 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916ca4b7-f10b-47c0-b576-4f2371a552a0" path="/var/lib/kubelet/pods/916ca4b7-f10b-47c0-b576-4f2371a552a0/volumes" Jul 06 23:49:47.118164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:47.118125 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:47.118611 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:47.118192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:47.140801 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:47.140768 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:47.777154 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:47.777125 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hdjb4" Jul 06 23:49:59.912668 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:49:59.912603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:00.523958 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:00.523922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:03.740097 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.740065 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-r5t95"] Jul 06 23:50:03.740484 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.740355 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916ca4b7-f10b-47c0-b576-4f2371a552a0" containerName="registry-server" Jul 06 23:50:03.740484 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.740365 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="916ca4b7-f10b-47c0-b576-4f2371a552a0" containerName="registry-server" Jul 06 23:50:03.740484 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.740425 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="916ca4b7-f10b-47c0-b576-4f2371a552a0" containerName="registry-server" Jul 06 23:50:03.747897 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.747874 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" Jul 06 23:50:03.750351 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.750328 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l7mfq\"" Jul 06 23:50:03.753015 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.752994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-r5t95"] Jul 06 23:50:03.835343 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.835304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkvz4\" (UniqueName: \"kubernetes.io/projected/d1d94964-164e-449e-b637-2a683b6d6620-kube-api-access-nkvz4\") pod \"authorino-operator-75cc8d5cb7-r5t95\" (UID: \"d1d94964-164e-449e-b637-2a683b6d6620\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" Jul 06 23:50:03.935874 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.935844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkvz4\" (UniqueName: \"kubernetes.io/projected/d1d94964-164e-449e-b637-2a683b6d6620-kube-api-access-nkvz4\") pod \"authorino-operator-75cc8d5cb7-r5t95\" (UID: \"d1d94964-164e-449e-b637-2a683b6d6620\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" Jul 06 23:50:03.951389 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:03.951353 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkvz4\" (UniqueName: \"kubernetes.io/projected/d1d94964-164e-449e-b637-2a683b6d6620-kube-api-access-nkvz4\") pod \"authorino-operator-75cc8d5cb7-r5t95\" (UID: \"d1d94964-164e-449e-b637-2a683b6d6620\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" Jul 06 23:50:04.059099 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:04.059067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" Jul 06 23:50:04.198414 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:04.198364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-r5t95"] Jul 06 23:50:04.201441 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:50:04.201412 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d94964_164e_449e_b637_2a683b6d6620.slice/crio-b3644d4c63ea27d03132ab39f3e3a4ada80e0143f33ae3d22c12d44ca5284527 WatchSource:0}: Error finding container b3644d4c63ea27d03132ab39f3e3a4ada80e0143f33ae3d22c12d44ca5284527: Status 404 returned error can't find the container with id b3644d4c63ea27d03132ab39f3e3a4ada80e0143f33ae3d22c12d44ca5284527 Jul 06 23:50:04.710698 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:04.710658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:04.831805 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:04.831732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" event={"ID":"d1d94964-164e-449e-b637-2a683b6d6620","Type":"ContainerStarted","Data":"b3644d4c63ea27d03132ab39f3e3a4ada80e0143f33ae3d22c12d44ca5284527"} Jul 06 23:50:06.537893 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:06.537854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:06.637986 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:06.637890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:06.842110 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:06.842072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" event={"ID":"d1d94964-164e-449e-b637-2a683b6d6620","Type":"ContainerStarted","Data":"f2ecf241ca2cea3da3290c73b5c364997aa62cdd8dce5aac86bdbac4a15413e9"} Jul 06 23:50:06.842300 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:06.842135 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" Jul 06 23:50:06.860803 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:06.860747 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" podStartSLOduration=1.7585321710000001 podStartE2EDuration="3.860733852s" totalImagesPullingTime="2.102201681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:06.859371289 +0000 UTC m=+436.886276517" watchObservedRunningTime="2026-07-06 23:50:06.860733852 +0000 UTC m=+436.887639081" Jul 06 23:50:09.241003 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:09.240970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:09.919907 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:09.919871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:12.510272 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:12.510232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:12.749546 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:12.749468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:15.154918 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:15.154881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:16.322977 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:16.322933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:16.423020 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:16.422984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:16.700787 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:16.700701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:16.801949 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:16.801916 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:17.848296 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:17.848267 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r5t95" Jul 06 23:50:17.973902 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:17.973865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:18.062708 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:18.062675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:20.518401 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:20.518366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:20.661107 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:20.661062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:20.732668 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:20.732609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:27.476451 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:27.476417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:27.579679 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:27.579637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:28.782292 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.782259 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd"] Jul 06 23:50:28.785559 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.785543 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:28.788271 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.788254 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z9kl7\"" Jul 06 23:50:28.799091 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.799071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd"] Jul 06 23:50:28.850419 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.850381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68908246-04d4-4590-b943-8c0d908eb1de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:28.850553 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.850440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl9wl\" (UniqueName: \"kubernetes.io/projected/68908246-04d4-4590-b943-8c0d908eb1de-kube-api-access-bl9wl\") pod \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:28.951047 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.951009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68908246-04d4-4590-b943-8c0d908eb1de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:28.951241 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.951064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl9wl\" (UniqueName: \"kubernetes.io/projected/68908246-04d4-4590-b943-8c0d908eb1de-kube-api-access-bl9wl\") pod \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:28.951474 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.951452 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68908246-04d4-4590-b943-8c0d908eb1de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:28.962986 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:28.962959 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl9wl\" (UniqueName: \"kubernetes.io/projected/68908246-04d4-4590-b943-8c0d908eb1de-kube-api-access-bl9wl\") pod \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:29.059465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.059379 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd"] Jul 06 23:50:29.059658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.059645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:29.063802 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.063774 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd"] Jul 06 23:50:29.094857 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.094820 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg"] Jul 06 23:50:29.098138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.098117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.132124 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.132095 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg"] Jul 06 23:50:29.152554 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.152517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krptb\" (UniqueName: \"kubernetes.io/projected/11748037-945e-4a89-814f-0ee228786795-kube-api-access-krptb\") pod \"kuadrant-operator-controller-manager-55467d9f7-x6ddg\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.152736 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.152588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11748037-945e-4a89-814f-0ee228786795-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-x6ddg\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.253596 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.253559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krptb\" (UniqueName: \"kubernetes.io/projected/11748037-945e-4a89-814f-0ee228786795-kube-api-access-krptb\") pod \"kuadrant-operator-controller-manager-55467d9f7-x6ddg\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.253786 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.253608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11748037-945e-4a89-814f-0ee228786795-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-x6ddg\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.253979 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.253961 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11748037-945e-4a89-814f-0ee228786795-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-x6ddg\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.277024 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.276996 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krptb\" (UniqueName: \"kubernetes.io/projected/11748037-945e-4a89-814f-0ee228786795-kube-api-access-krptb\") pod \"kuadrant-operator-controller-manager-55467d9f7-x6ddg\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.408433 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.408341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:29.548465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.548401 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg"] Jul 06 23:50:29.550939 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:50:29.550914 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11748037_945e_4a89_814f_0ee228786795.slice/crio-d68239507b73885c8f46840fd32a5bb8ac8badfe3095dc03e5e4ee08fa84fc29 WatchSource:0}: Error finding container d68239507b73885c8f46840fd32a5bb8ac8badfe3095dc03e5e4ee08fa84fc29: Status 404 returned error can't find the container with id d68239507b73885c8f46840fd32a5bb8ac8badfe3095dc03e5e4ee08fa84fc29 Jul 06 23:50:29.944362 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:29.944321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" event={"ID":"11748037-945e-4a89-814f-0ee228786795","Type":"ContainerStarted","Data":"d68239507b73885c8f46840fd32a5bb8ac8badfe3095dc03e5e4ee08fa84fc29"} Jul 06 23:50:31.602741 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:50:31.602701 2564 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-s5jmd_kuadrant-system_68908246-04d4-4590-b943-8c0d908eb1de_0(00939f6661e537c1ecfe46980a785b2f53e6d20eb8bf7afb74dd7a35f19b6c3f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-s5jmd to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 06 23:50:31.603007 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:50:31.602769 2564 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-s5jmd_kuadrant-system_68908246-04d4-4590-b943-8c0d908eb1de_0(00939f6661e537c1ecfe46980a785b2f53e6d20eb8bf7afb74dd7a35f19b6c3f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-s5jmd to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:31.956430 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:31.956340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:31.964169 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:31.964142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:31.966902 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:50:31.966873 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" podUID="68908246-04d4-4590-b943-8c0d908eb1de" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:32.078278 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.078240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl9wl\" (UniqueName: \"kubernetes.io/projected/68908246-04d4-4590-b943-8c0d908eb1de-kube-api-access-bl9wl\") pod \"68908246-04d4-4590-b943-8c0d908eb1de\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " Jul 06 23:50:32.078464 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.078296 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68908246-04d4-4590-b943-8c0d908eb1de-extensions-socket-volume\") pod \"68908246-04d4-4590-b943-8c0d908eb1de\" (UID: \"68908246-04d4-4590-b943-8c0d908eb1de\") " Jul 06 23:50:32.078666 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.078613 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68908246-04d4-4590-b943-8c0d908eb1de-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "68908246-04d4-4590-b943-8c0d908eb1de" (UID: "68908246-04d4-4590-b943-8c0d908eb1de"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:32.080805 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.080776 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68908246-04d4-4590-b943-8c0d908eb1de-kube-api-access-bl9wl" (OuterVolumeSpecName: "kube-api-access-bl9wl") pod "68908246-04d4-4590-b943-8c0d908eb1de" (UID: "68908246-04d4-4590-b943-8c0d908eb1de"). InnerVolumeSpecName "kube-api-access-bl9wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:32.179319 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.179285 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl9wl\" (UniqueName: \"kubernetes.io/projected/68908246-04d4-4590-b943-8c0d908eb1de-kube-api-access-bl9wl\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:50:32.179319 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.179315 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68908246-04d4-4590-b943-8c0d908eb1de-extensions-socket-volume\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:50:32.294399 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.294364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:32.418256 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.418171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:32.655414 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.655320 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68908246-04d4-4590-b943-8c0d908eb1de" path="/var/lib/kubelet/pods/68908246-04d4-4590-b943-8c0d908eb1de/volumes" Jul 06 23:50:32.961029 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:32.960955 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:32.965932 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:50:32.965896 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" podUID="68908246-04d4-4590-b943-8c0d908eb1de" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:33.966196 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:33.966162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" event={"ID":"11748037-945e-4a89-814f-0ee228786795","Type":"ContainerStarted","Data":"6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4"} Jul 06 23:50:33.966568 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:33.966241 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:33.968204 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:50:33.968183 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-s5jmd\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" podUID="68908246-04d4-4590-b943-8c0d908eb1de" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s5jmd" Jul 06 23:50:33.988820 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:33.988774 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" podStartSLOduration=1.056852975 podStartE2EDuration="4.98876014s" totalImagesPullingTime="3.931907165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:33.986644526 +0000 UTC m=+464.013549746" watchObservedRunningTime="2026-07-06 23:50:33.98876014 +0000 UTC m=+464.015665368" Jul 06 23:50:35.911112 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:35.911079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:35.965564 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:35.965525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:36.131107 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:36.131069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:36.192234 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:36.192146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:38.671726 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:38.671689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:38.772776 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:38.772735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:44.973042 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:44.973012 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:45.160475 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:45.160439 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:45.263920 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:45.263891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:49.899548 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:49.899519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:50.001404 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:50.001372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:51.059462 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:51.059427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:51.159743 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:51.159706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:53.140441 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:53.140404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:53.198653 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:53.198592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:53.693084 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:53.693040 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:53.745565 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:53.745530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:54.305164 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:54.305120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:54.421093 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:54.421058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:57.147293 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.147264 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg"] Jul 06 23:50:57.147671 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.147456 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" podUID="11748037-945e-4a89-814f-0ee228786795" containerName="manager" containerID="cri-o://6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4" gracePeriod=10 Jul 06 23:50:57.408118 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.408054 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:57.590875 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.590841 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krptb\" (UniqueName: \"kubernetes.io/projected/11748037-945e-4a89-814f-0ee228786795-kube-api-access-krptb\") pod \"11748037-945e-4a89-814f-0ee228786795\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " Jul 06 23:50:57.591028 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.590902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11748037-945e-4a89-814f-0ee228786795-extensions-socket-volume\") pod \"11748037-945e-4a89-814f-0ee228786795\" (UID: \"11748037-945e-4a89-814f-0ee228786795\") " Jul 06 23:50:57.591265 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.591242 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11748037-945e-4a89-814f-0ee228786795-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "11748037-945e-4a89-814f-0ee228786795" (UID: "11748037-945e-4a89-814f-0ee228786795"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:57.592944 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.592917 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11748037-945e-4a89-814f-0ee228786795-kube-api-access-krptb" (OuterVolumeSpecName: "kube-api-access-krptb") pod "11748037-945e-4a89-814f-0ee228786795" (UID: "11748037-945e-4a89-814f-0ee228786795"). InnerVolumeSpecName "kube-api-access-krptb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:57.692549 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.692446 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11748037-945e-4a89-814f-0ee228786795-extensions-socket-volume\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:50:57.692549 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:57.692495 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krptb\" (UniqueName: \"kubernetes.io/projected/11748037-945e-4a89-814f-0ee228786795-kube-api-access-krptb\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:50:58.076570 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.076538 2564 generic.go:356] "Generic (PLEG): container finished" podID="11748037-945e-4a89-814f-0ee228786795" containerID="6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4" exitCode=0 Jul 06 23:50:58.076757 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.076585 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" Jul 06 23:50:58.076757 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.076622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" event={"ID":"11748037-945e-4a89-814f-0ee228786795","Type":"ContainerDied","Data":"6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4"} Jul 06 23:50:58.076757 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.076692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg" event={"ID":"11748037-945e-4a89-814f-0ee228786795","Type":"ContainerDied","Data":"d68239507b73885c8f46840fd32a5bb8ac8badfe3095dc03e5e4ee08fa84fc29"} Jul 06 23:50:58.076757 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.076715 2564 scope.go:117] "RemoveContainer" containerID="6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4" Jul 06 23:50:58.086542 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.086524 2564 scope.go:117] "RemoveContainer" containerID="6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4" Jul 06 23:50:58.086821 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:50:58.086799 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4\": container with ID starting with 6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4 not found: ID does not exist" containerID="6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4" Jul 06 23:50:58.086874 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.086830 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4"} err="failed to get container status \"6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4\": rpc error: code = NotFound desc = could not find container \"6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4\": container with ID starting with 6f477d05dbf5dcecd91a5c1daaa3ee89e489489958670704744290b8aea05fa4 not found: ID does not exist" Jul 06 23:50:58.103335 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.103309 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg"] Jul 06 23:50:58.108945 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.108925 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x6ddg"] Jul 06 23:50:58.115684 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.115650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:58.181892 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.181864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:50:58.654143 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:50:58.654114 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11748037-945e-4a89-814f-0ee228786795" path="/var/lib/kubelet/pods/11748037-945e-4a89-814f-0ee228786795/volumes" Jul 06 23:51:01.443261 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:01.443231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:01.541696 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:01.541656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:04.177462 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:04.177420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:04.253893 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:04.253843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:04.349923 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:04.349881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:04.450930 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:04.450839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:04.552237 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:04.552202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:08.906893 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:08.906852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:09.015256 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:09.015219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:18.222772 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.222734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:18.284066 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.284035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:18.426672 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.426620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:18.430097 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.430069 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfnr6"] Jul 06 23:51:18.430387 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.430376 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11748037-945e-4a89-814f-0ee228786795" containerName="manager" Jul 06 23:51:18.430434 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.430389 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="11748037-945e-4a89-814f-0ee228786795" containerName="manager" Jul 06 23:51:18.430474 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.430460 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="11748037-945e-4a89-814f-0ee228786795" containerName="manager" Jul 06 23:51:18.432506 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.432490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.434767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.434731 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 23:51:18.434892 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.434814 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c6m7v\"" Jul 06 23:51:18.442927 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.442902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfnr6"] Jul 06 23:51:18.534011 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.533975 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfnr6"] Jul 06 23:51:18.535344 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.535304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:18.560592 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.560562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c620ce72-372f-4458-b89e-8ab6b71197fa-config-file\") pod \"limitador-limitador-64c94fd64f-cfnr6\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.560754 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.560678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9rr\" (UniqueName: \"kubernetes.io/projected/c620ce72-372f-4458-b89e-8ab6b71197fa-kube-api-access-vv9rr\") pod \"limitador-limitador-64c94fd64f-cfnr6\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.633136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.633102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:18.661873 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.661843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c620ce72-372f-4458-b89e-8ab6b71197fa-config-file\") pod \"limitador-limitador-64c94fd64f-cfnr6\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.662045 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.661917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9rr\" (UniqueName: \"kubernetes.io/projected/c620ce72-372f-4458-b89e-8ab6b71197fa-kube-api-access-vv9rr\") pod \"limitador-limitador-64c94fd64f-cfnr6\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.662520 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.662495 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c620ce72-372f-4458-b89e-8ab6b71197fa-config-file\") pod \"limitador-limitador-64c94fd64f-cfnr6\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.669953 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.669931 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9rr\" (UniqueName: \"kubernetes.io/projected/c620ce72-372f-4458-b89e-8ab6b71197fa-kube-api-access-vv9rr\") pod \"limitador-limitador-64c94fd64f-cfnr6\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.743471 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.743435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:18.878294 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:18.878266 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfnr6"] Jul 06 23:51:18.881026 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:51:18.881000 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc620ce72_372f_4458_b89e_8ab6b71197fa.slice/crio-2c808f2bd25189a0128b9931db6d259a43d49232bba5234102d94f11416f6cf1 WatchSource:0}: Error finding container 2c808f2bd25189a0128b9931db6d259a43d49232bba5234102d94f11416f6cf1: Status 404 returned error can't find the container with id 2c808f2bd25189a0128b9931db6d259a43d49232bba5234102d94f11416f6cf1 Jul 06 23:51:19.173089 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:19.173000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" event={"ID":"c620ce72-372f-4458-b89e-8ab6b71197fa","Type":"ContainerStarted","Data":"2c808f2bd25189a0128b9931db6d259a43d49232bba5234102d94f11416f6cf1"} Jul 06 23:51:19.276647 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:19.276593 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:19.375561 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:19.375504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:22.189645 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:22.189595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" event={"ID":"c620ce72-372f-4458-b89e-8ab6b71197fa","Type":"ContainerStarted","Data":"9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a"} Jul 06 23:51:22.190065 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:22.189685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:22.205948 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:22.205890 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" podStartSLOduration=1.780392566 podStartE2EDuration="4.205872058s" totalImagesPullingTime="2.425479492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:51:22.205344693 +0000 UTC m=+512.232249934" watchObservedRunningTime="2026-07-06 23:51:22.205872058 +0000 UTC m=+512.232777286" Jul 06 23:51:33.194920 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.194889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:33.340582 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.340544 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfnr6"] Jul 06 23:51:33.340785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.340740 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" podUID="c620ce72-372f-4458-b89e-8ab6b71197fa" containerName="limitador" containerID="cri-o://9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a" gracePeriod=30 Jul 06 23:51:33.897794 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.897768 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:33.992951 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.992849 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv9rr\" (UniqueName: \"kubernetes.io/projected/c620ce72-372f-4458-b89e-8ab6b71197fa-kube-api-access-vv9rr\") pod \"c620ce72-372f-4458-b89e-8ab6b71197fa\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " Jul 06 23:51:33.992951 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.992932 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c620ce72-372f-4458-b89e-8ab6b71197fa-config-file\") pod \"c620ce72-372f-4458-b89e-8ab6b71197fa\" (UID: \"c620ce72-372f-4458-b89e-8ab6b71197fa\") " Jul 06 23:51:33.993300 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.993277 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c620ce72-372f-4458-b89e-8ab6b71197fa-config-file" (OuterVolumeSpecName: "config-file") pod "c620ce72-372f-4458-b89e-8ab6b71197fa" (UID: "c620ce72-372f-4458-b89e-8ab6b71197fa"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:33.994941 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:33.994918 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c620ce72-372f-4458-b89e-8ab6b71197fa-kube-api-access-vv9rr" (OuterVolumeSpecName: "kube-api-access-vv9rr") pod "c620ce72-372f-4458-b89e-8ab6b71197fa" (UID: "c620ce72-372f-4458-b89e-8ab6b71197fa"). InnerVolumeSpecName "kube-api-access-vv9rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:34.093950 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.093913 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv9rr\" (UniqueName: \"kubernetes.io/projected/c620ce72-372f-4458-b89e-8ab6b71197fa-kube-api-access-vv9rr\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:51:34.093950 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.093942 2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c620ce72-372f-4458-b89e-8ab6b71197fa-config-file\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:51:34.250304 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.250218 2564 generic.go:356] "Generic (PLEG): container finished" podID="c620ce72-372f-4458-b89e-8ab6b71197fa" containerID="9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a" exitCode=0 Jul 06 23:51:34.250304 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.250264 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" Jul 06 23:51:34.250827 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.250308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" event={"ID":"c620ce72-372f-4458-b89e-8ab6b71197fa","Type":"ContainerDied","Data":"9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a"} Jul 06 23:51:34.250827 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.250365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfnr6" event={"ID":"c620ce72-372f-4458-b89e-8ab6b71197fa","Type":"ContainerDied","Data":"2c808f2bd25189a0128b9931db6d259a43d49232bba5234102d94f11416f6cf1"} Jul 06 23:51:34.250827 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.250397 2564 scope.go:117] "RemoveContainer" containerID="9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a" Jul 06 23:51:34.262621 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.262598 2564 scope.go:117] "RemoveContainer" containerID="9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a" Jul 06 23:51:34.263087 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:34.262962 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a\": container with ID starting with 9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a not found: ID does not exist" containerID="9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a" Jul 06 23:51:34.263087 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.262995 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a"} err="failed to get container status \"9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a\": rpc error: code = NotFound desc = could not find container \"9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a\": container with ID starting with 9f049f301517111cf8d78b5ca6f99984cf0d22d7566abe73c6493252d5ac377a not found: ID does not exist" Jul 06 23:51:34.276004 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.275976 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfnr6"] Jul 06 23:51:34.280221 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.280200 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfnr6"] Jul 06 23:51:34.653876 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:34.653841 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c620ce72-372f-4458-b89e-8ab6b71197fa" path="/var/lib/kubelet/pods/c620ce72-372f-4458-b89e-8ab6b71197fa/volumes" Jul 06 23:51:35.647320 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.647286 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-rwzgz"] Jul 06 23:51:35.647821 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.647801 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c620ce72-372f-4458-b89e-8ab6b71197fa" containerName="limitador" Jul 06 23:51:35.647876 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.647826 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c620ce72-372f-4458-b89e-8ab6b71197fa" containerName="limitador" Jul 06 23:51:35.647925 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.647914 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c620ce72-372f-4458-b89e-8ab6b71197fa" containerName="limitador" Jul 06 23:51:35.651280 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.651256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:35.653732 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.653712 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 23:51:35.653838 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.653772 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fs2gb\"" Jul 06 23:51:35.657168 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.657147 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-rwzgz"] Jul 06 23:51:35.686230 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.686200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:35.807710 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.807672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c56466fa-f92c-42f2-9756-494d2003a9d6-data\") pod \"postgres-69fb4594cf-rwzgz\" (UID: \"c56466fa-f92c-42f2-9756-494d2003a9d6\") " pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:35.807901 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.807753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wr2\" (UniqueName: \"kubernetes.io/projected/c56466fa-f92c-42f2-9756-494d2003a9d6-kube-api-access-d5wr2\") pod \"postgres-69fb4594cf-rwzgz\" (UID: \"c56466fa-f92c-42f2-9756-494d2003a9d6\") " pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:35.909261 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.909163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c56466fa-f92c-42f2-9756-494d2003a9d6-data\") pod \"postgres-69fb4594cf-rwzgz\" (UID: \"c56466fa-f92c-42f2-9756-494d2003a9d6\") " pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:35.909261 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.909202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wr2\" (UniqueName: \"kubernetes.io/projected/c56466fa-f92c-42f2-9756-494d2003a9d6-kube-api-access-d5wr2\") pod \"postgres-69fb4594cf-rwzgz\" (UID: \"c56466fa-f92c-42f2-9756-494d2003a9d6\") " pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:35.909667 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.909610 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c56466fa-f92c-42f2-9756-494d2003a9d6-data\") pod \"postgres-69fb4594cf-rwzgz\" (UID: \"c56466fa-f92c-42f2-9756-494d2003a9d6\") " pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:35.918258 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.918231 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wr2\" (UniqueName: \"kubernetes.io/projected/c56466fa-f92c-42f2-9756-494d2003a9d6-kube-api-access-d5wr2\") pod \"postgres-69fb4594cf-rwzgz\" (UID: \"c56466fa-f92c-42f2-9756-494d2003a9d6\") " pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:35.962233 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:35.962186 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:36.091976 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:36.091951 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-rwzgz"] Jul 06 23:51:36.094396 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:51:36.094356 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56466fa_f92c_42f2_9756_494d2003a9d6.slice/crio-6a1b427053de9627a6c40caa76140f91e984bee87293e948069449623f388c69 WatchSource:0}: Error finding container 6a1b427053de9627a6c40caa76140f91e984bee87293e948069449623f388c69: Status 404 returned error can't find the container with id 6a1b427053de9627a6c40caa76140f91e984bee87293e948069449623f388c69 Jul 06 23:51:36.260427 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:36.260335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-rwzgz" event={"ID":"c56466fa-f92c-42f2-9756-494d2003a9d6","Type":"ContainerStarted","Data":"6a1b427053de9627a6c40caa76140f91e984bee87293e948069449623f388c69"} Jul 06 23:51:41.286944 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:41.286900 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-rwzgz" event={"ID":"c56466fa-f92c-42f2-9756-494d2003a9d6","Type":"ContainerStarted","Data":"7740ae1bd33f4301d6f015857ea59401859e93ccf118e1e9214c5d95e300c7ad"} Jul 06 23:51:41.287451 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:41.286986 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:41.306987 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:41.306940 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-rwzgz" podStartSLOduration=1.8915852690000001 podStartE2EDuration="6.306923703s" totalImagesPullingTime="4.415338434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:51:41.304250995 +0000 UTC m=+531.331156225" watchObservedRunningTime="2026-07-06 23:51:41.306923703 +0000 UTC m=+531.333828935" Jul 06 23:51:47.321763 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:47.321693 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-rwzgz" Jul 06 23:51:47.716583 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:47.716494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:47.882548 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:47.882517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:47.918727 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:47.918526 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tdqxx"] Jul 06 23:51:47.934816 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:47.934718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tdqxx"] Jul 06 23:51:47.934987 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:47.934827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tdqxx" Jul 06 23:51:47.937437 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:47.937381 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sx2lw\"" Jul 06 23:51:48.110724 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.110685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f22cn\" (UniqueName: \"kubernetes.io/projected/50b099e4-cda6-4df8-9595-4708668b4f80-kube-api-access-f22cn\") pod \"authorino-5d4b67c446-tdqxx\" (UID: \"50b099e4-cda6-4df8-9595-4708668b4f80\") " pod="kuadrant-system/authorino-5d4b67c446-tdqxx" Jul 06 23:51:48.112212 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.112189 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tdqxx"] Jul 06 23:51:48.112435 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:48.112416 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-f22cn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-tdqxx" podUID="50b099e4-cda6-4df8-9595-4708668b4f80" Jul 06 23:51:48.212063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.212024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f22cn\" (UniqueName: \"kubernetes.io/projected/50b099e4-cda6-4df8-9595-4708668b4f80-kube-api-access-f22cn\") pod \"authorino-5d4b67c446-tdqxx\" (UID: \"50b099e4-cda6-4df8-9595-4708668b4f80\") " pod="kuadrant-system/authorino-5d4b67c446-tdqxx" Jul 06 23:51:48.225345 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.225306 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cd5bf955f-775lp"] Jul 06 23:51:48.228982 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.228955 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f22cn\" (UniqueName: \"kubernetes.io/projected/50b099e4-cda6-4df8-9595-4708668b4f80-kube-api-access-f22cn\") pod \"authorino-5d4b67c446-tdqxx\" (UID: \"50b099e4-cda6-4df8-9595-4708668b4f80\") " pod="kuadrant-system/authorino-5d4b67c446-tdqxx" Jul 06 23:51:48.250605 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.250573 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd5bf955f-775lp"] Jul 06 23:51:48.250802 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.250695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.253048 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.253026 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 23:51:48.318658 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.318595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tdqxx" Jul 06 23:51:48.337143 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.337103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tdqxx" Jul 06 23:51:48.413830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.413733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9dd945f-342b-469a-b26d-8706c457b160-tls-cert\") pod \"authorino-cd5bf955f-775lp\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.413830 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.413776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkvm\" (UniqueName: \"kubernetes.io/projected/c9dd945f-342b-469a-b26d-8706c457b160-kube-api-access-pfkvm\") pod \"authorino-cd5bf955f-775lp\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.514274 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.514237 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f22cn\" (UniqueName: \"kubernetes.io/projected/50b099e4-cda6-4df8-9595-4708668b4f80-kube-api-access-f22cn\") pod \"50b099e4-cda6-4df8-9595-4708668b4f80\" (UID: \"50b099e4-cda6-4df8-9595-4708668b4f80\") " Jul 06 23:51:48.514528 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.514510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9dd945f-342b-469a-b26d-8706c457b160-tls-cert\") pod \"authorino-cd5bf955f-775lp\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.514571 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.514550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkvm\" (UniqueName: \"kubernetes.io/projected/c9dd945f-342b-469a-b26d-8706c457b160-kube-api-access-pfkvm\") pod \"authorino-cd5bf955f-775lp\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.516442 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.516413 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b099e4-cda6-4df8-9595-4708668b4f80-kube-api-access-f22cn" (OuterVolumeSpecName: "kube-api-access-f22cn") pod "50b099e4-cda6-4df8-9595-4708668b4f80" (UID: "50b099e4-cda6-4df8-9595-4708668b4f80"). InnerVolumeSpecName "kube-api-access-f22cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:48.516952 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.516933 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9dd945f-342b-469a-b26d-8706c457b160-tls-cert\") pod \"authorino-cd5bf955f-775lp\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.522254 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.522233 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkvm\" (UniqueName: \"kubernetes.io/projected/c9dd945f-342b-469a-b26d-8706c457b160-kube-api-access-pfkvm\") pod \"authorino-cd5bf955f-775lp\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.560240 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.560189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:51:48.615988 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.615949 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f22cn\" (UniqueName: \"kubernetes.io/projected/50b099e4-cda6-4df8-9595-4708668b4f80-kube-api-access-f22cn\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:51:48.717404 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:48.717327 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd5bf955f-775lp"] Jul 06 23:51:48.720999 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:51:48.720973 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9dd945f_342b_469a_b26d_8706c457b160.slice/crio-823984bff7c8fe816ab96071998759bcd47730688733291993343bc8056fbabb WatchSource:0}: Error finding container 823984bff7c8fe816ab96071998759bcd47730688733291993343bc8056fbabb: Status 404 returned error can't find the container with id 823984bff7c8fe816ab96071998759bcd47730688733291993343bc8056fbabb Jul 06 23:51:49.324314 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:49.324278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tdqxx" Jul 06 23:51:49.324503 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:49.324271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd5bf955f-775lp" event={"ID":"c9dd945f-342b-469a-b26d-8706c457b160","Type":"ContainerStarted","Data":"823984bff7c8fe816ab96071998759bcd47730688733291993343bc8056fbabb"} Jul 06 23:51:49.356831 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:49.356797 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tdqxx"] Jul 06 23:51:49.360663 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:49.360622 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tdqxx"] Jul 06 23:51:50.656092 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:50.656053 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50b099e4-cda6-4df8-9595-4708668b4f80" path="/var/lib/kubelet/pods/50b099e4-cda6-4df8-9595-4708668b4f80/volumes" Jul 06 23:51:52.340421 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:52.340381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd5bf955f-775lp" event={"ID":"c9dd945f-342b-469a-b26d-8706c457b160","Type":"ContainerStarted","Data":"331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3"} Jul 06 23:51:57.414210 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.414138 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cd5bf955f-775lp" podStartSLOduration=6.155250285 podStartE2EDuration="9.414118809s" totalImagesPullingTime="3.258868524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:51:52.367494105 +0000 UTC m=+542.394399334" watchObservedRunningTime="2026-07-06 23:51:57.414118809 +0000 UTC m=+547.441024083" Jul 06 23:51:57.415184 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.415160 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7576cdb7d5-2m8j8"] Jul 06 23:51:57.418204 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.418188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:57.422436 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.422413 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddfvt\"" Jul 06 23:51:57.422624 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.422602 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 23:51:57.422745 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.422672 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 23:51:57.443483 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.443438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:57.443677 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.443594 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7576cdb7d5-2m8j8"] Jul 06 23:51:57.488424 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.488393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:57.488591 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.488473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bln5p\" (UniqueName: \"kubernetes.io/projected/2fea30e2-73af-4061-ad39-a24c6d9e1626-kube-api-access-bln5p\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:57.543762 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.543720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:57.589399 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.589361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bln5p\" (UniqueName: \"kubernetes.io/projected/2fea30e2-73af-4061-ad39-a24c6d9e1626-kube-api-access-bln5p\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:57.589566 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.589423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:57.589566 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:57.589531 2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 23:51:57.589660 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:57.589596 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert podName:2fea30e2-73af-4061-ad39-a24c6d9e1626 nodeName:}" failed. No retries permitted until 2026-07-06 23:51:58.089575135 +0000 UTC m=+548.116480349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert") pod "maas-controller-7576cdb7d5-2m8j8" (UID: "2fea30e2-73af-4061-ad39-a24c6d9e1626") : secret "maas-controller-webhook-cert" not found Jul 06 23:51:57.619288 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.619263 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bln5p\" (UniqueName: \"kubernetes.io/projected/2fea30e2-73af-4061-ad39-a24c6d9e1626-kube-api-access-bln5p\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:57.629043 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.629017 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-jbrsr"] Jul 06 23:51:57.631786 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.631769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:57.657726 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.657692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-jbrsr"] Jul 06 23:51:57.690465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.690385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert\") pod \"maas-controller-696cfbbf9b-jbrsr\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:57.690465 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.690445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmfx\" (UniqueName: \"kubernetes.io/projected/e16cb968-e8b3-4100-b784-dfa8fbad9722-kube-api-access-npmfx\") pod \"maas-controller-696cfbbf9b-jbrsr\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:57.790859 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.790827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert\") pod \"maas-controller-696cfbbf9b-jbrsr\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:57.791021 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.790883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmfx\" (UniqueName: \"kubernetes.io/projected/e16cb968-e8b3-4100-b784-dfa8fbad9722-kube-api-access-npmfx\") pod \"maas-controller-696cfbbf9b-jbrsr\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:57.791021 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:57.790970 2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 23:51:57.791115 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:57.791032 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert podName:e16cb968-e8b3-4100-b784-dfa8fbad9722 nodeName:}" failed. No retries permitted until 2026-07-06 23:51:58.291016917 +0000 UTC m=+548.317922124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert") pod "maas-controller-696cfbbf9b-jbrsr" (UID: "e16cb968-e8b3-4100-b784-dfa8fbad9722") : secret "maas-controller-webhook-cert" not found Jul 06 23:51:57.809067 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:57.809031 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmfx\" (UniqueName: \"kubernetes.io/projected/e16cb968-e8b3-4100-b784-dfa8fbad9722-kube-api-access-npmfx\") pod \"maas-controller-696cfbbf9b-jbrsr\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:58.092759 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:58.092717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:58.092968 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:58.092857 2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 23:51:58.092968 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:51:58.092921 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert podName:2fea30e2-73af-4061-ad39-a24c6d9e1626 nodeName:}" failed. No retries permitted until 2026-07-06 23:51:59.09290651 +0000 UTC m=+549.119811716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert") pod "maas-controller-7576cdb7d5-2m8j8" (UID: "2fea30e2-73af-4061-ad39-a24c6d9e1626") : secret "maas-controller-webhook-cert" not found Jul 06 23:51:58.294640 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:58.294602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert\") pod \"maas-controller-696cfbbf9b-jbrsr\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:58.297117 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:58.297086 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert\") pod \"maas-controller-696cfbbf9b-jbrsr\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:58.307951 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:58.307924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:51:58.542169 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:58.542132 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:51:58.687831 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:58.687804 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-jbrsr"] Jul 06 23:51:58.690302 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:51:58.690270 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16cb968_e8b3_4100_b784_dfa8fbad9722.slice/crio-5a781e358fc1a4ef73d881402a6ccbcfe8981c51b08d79d498524779bd219db9 WatchSource:0}: Error finding container 5a781e358fc1a4ef73d881402a6ccbcfe8981c51b08d79d498524779bd219db9: Status 404 returned error can't find the container with id 5a781e358fc1a4ef73d881402a6ccbcfe8981c51b08d79d498524779bd219db9 Jul 06 23:51:59.101685 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:59.101649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:59.104216 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:59.104190 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert\") pod \"maas-controller-7576cdb7d5-2m8j8\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:59.230759 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:59.230722 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:51:59.375163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:59.375064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" event={"ID":"e16cb968-e8b3-4100-b784-dfa8fbad9722","Type":"ContainerStarted","Data":"5a781e358fc1a4ef73d881402a6ccbcfe8981c51b08d79d498524779bd219db9"} Jul 06 23:51:59.399754 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:51:59.399721 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7576cdb7d5-2m8j8"] Jul 06 23:51:59.402262 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:51:59.402221 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fea30e2_73af_4061_ad39_a24c6d9e1626.slice/crio-bc02ff03248d966d4cbd050be6660756132475c9eeef42ff0dbfcbffe09ded50 WatchSource:0}: Error finding container bc02ff03248d966d4cbd050be6660756132475c9eeef42ff0dbfcbffe09ded50: Status 404 returned error can't find the container with id bc02ff03248d966d4cbd050be6660756132475c9eeef42ff0dbfcbffe09ded50 Jul 06 23:52:00.384917 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:00.384865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" event={"ID":"2fea30e2-73af-4061-ad39-a24c6d9e1626","Type":"ContainerStarted","Data":"bc02ff03248d966d4cbd050be6660756132475c9eeef42ff0dbfcbffe09ded50"} Jul 06 23:52:01.564290 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:01.564198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:01.727162 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:01.727113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:02.397904 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:02.397862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" event={"ID":"e16cb968-e8b3-4100-b784-dfa8fbad9722","Type":"ContainerStarted","Data":"46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041"} Jul 06 23:52:02.398101 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:02.397964 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:52:02.399291 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:02.399263 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" event={"ID":"2fea30e2-73af-4061-ad39-a24c6d9e1626","Type":"ContainerStarted","Data":"a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d"} Jul 06 23:52:02.399412 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:02.399397 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:52:02.419668 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:02.419601 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" podStartSLOduration=2.806460965 podStartE2EDuration="5.419585436s" totalImagesPullingTime="2.613124471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:52:02.418390988 +0000 UTC m=+552.445296219" watchObservedRunningTime="2026-07-06 23:52:02.419585436 +0000 UTC m=+552.446490698" Jul 06 23:52:02.445226 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:02.445170 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" podStartSLOduration=3.5382692860000002 podStartE2EDuration="5.445151101s" totalImagesPullingTime="1.906881815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:52:02.444374757 +0000 UTC m=+552.471279986" watchObservedRunningTime="2026-07-06 23:52:02.445151101 +0000 UTC m=+552.472056329" Jul 06 23:52:13.411117 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.411084 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:52:13.411589 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.411338 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:52:13.493768 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.493730 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7576cdb7d5-2m8j8"] Jul 06 23:52:13.493995 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.493962 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" podUID="2fea30e2-73af-4061-ad39-a24c6d9e1626" containerName="manager" containerID="cri-o://a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d" gracePeriod=30 Jul 06 23:52:13.742361 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.742334 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:52:13.831723 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.831680 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert\") pod \"2fea30e2-73af-4061-ad39-a24c6d9e1626\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " Jul 06 23:52:13.831874 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.831749 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bln5p\" (UniqueName: \"kubernetes.io/projected/2fea30e2-73af-4061-ad39-a24c6d9e1626-kube-api-access-bln5p\") pod \"2fea30e2-73af-4061-ad39-a24c6d9e1626\" (UID: \"2fea30e2-73af-4061-ad39-a24c6d9e1626\") " Jul 06 23:52:13.833940 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.833901 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2fea30e2-73af-4061-ad39-a24c6d9e1626" (UID: "2fea30e2-73af-4061-ad39-a24c6d9e1626"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:52:13.834055 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.834005 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fea30e2-73af-4061-ad39-a24c6d9e1626-kube-api-access-bln5p" (OuterVolumeSpecName: "kube-api-access-bln5p") pod "2fea30e2-73af-4061-ad39-a24c6d9e1626" (UID: "2fea30e2-73af-4061-ad39-a24c6d9e1626"). InnerVolumeSpecName "kube-api-access-bln5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:52:13.932980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.932874 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fea30e2-73af-4061-ad39-a24c6d9e1626-webhook-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:52:13.932980 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:13.932920 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bln5p\" (UniqueName: \"kubernetes.io/projected/2fea30e2-73af-4061-ad39-a24c6d9e1626-kube-api-access-bln5p\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:52:14.456444 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.456410 2564 generic.go:356] "Generic (PLEG): container finished" podID="2fea30e2-73af-4061-ad39-a24c6d9e1626" containerID="a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d" exitCode=0 Jul 06 23:52:14.456860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.456470 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" Jul 06 23:52:14.456860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.456491 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" event={"ID":"2fea30e2-73af-4061-ad39-a24c6d9e1626","Type":"ContainerDied","Data":"a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d"} Jul 06 23:52:14.456860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.456578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7576cdb7d5-2m8j8" event={"ID":"2fea30e2-73af-4061-ad39-a24c6d9e1626","Type":"ContainerDied","Data":"bc02ff03248d966d4cbd050be6660756132475c9eeef42ff0dbfcbffe09ded50"} Jul 06 23:52:14.456860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.456607 2564 scope.go:117] "RemoveContainer" containerID="a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d" Jul 06 23:52:14.465678 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.465659 2564 scope.go:117] "RemoveContainer" containerID="a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d" Jul 06 23:52:14.466004 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:52:14.465980 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d\": container with ID starting with a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d not found: ID does not exist" containerID="a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d" Jul 06 23:52:14.466052 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.466014 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d"} err="failed to get container status \"a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d\": rpc error: code = NotFound desc = could not find container \"a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d\": container with ID starting with a1d7545fd353eaa08e5a9d806c154293e0503dfeadeb202d763088c2385daf4d not found: ID does not exist" Jul 06 23:52:14.484645 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.484589 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7576cdb7d5-2m8j8"] Jul 06 23:52:14.496823 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.496790 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7576cdb7d5-2m8j8"] Jul 06 23:52:14.654815 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:14.654783 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fea30e2-73af-4061-ad39-a24c6d9e1626" path="/var/lib/kubelet/pods/2fea30e2-73af-4061-ad39-a24c6d9e1626/volumes" Jul 06 23:52:34.202795 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.202755 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:34.684686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.684647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:34.799968 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.799934 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5ddd4945c8-52wgr"] Jul 06 23:52:34.800350 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.800322 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fea30e2-73af-4061-ad39-a24c6d9e1626" containerName="manager" Jul 06 23:52:34.800350 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.800340 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fea30e2-73af-4061-ad39-a24c6d9e1626" containerName="manager" Jul 06 23:52:34.800553 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.800438 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fea30e2-73af-4061-ad39-a24c6d9e1626" containerName="manager" Jul 06 23:52:34.806767 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.806746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:34.809057 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.809034 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 23:52:34.847288 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.847250 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5ddd4945c8-52wgr"] Jul 06 23:52:34.901686 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.901651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th2s\" (UniqueName: \"kubernetes.io/projected/4cf12e6e-add2-4904-97a1-32e9f5db5af2-kube-api-access-5th2s\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:34.901860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.901718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cf12e6e-add2-4904-97a1-32e9f5db5af2-maas-api-tls\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:34.901860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.901756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf12e6e-add2-4904-97a1-32e9f5db5af2-openshift-service-ca\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:34.935421 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:34.935332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:35.002556 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.002510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cf12e6e-add2-4904-97a1-32e9f5db5af2-maas-api-tls\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:35.002556 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.002563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf12e6e-add2-4904-97a1-32e9f5db5af2-openshift-service-ca\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:35.002805 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.002592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th2s\" (UniqueName: \"kubernetes.io/projected/4cf12e6e-add2-4904-97a1-32e9f5db5af2-kube-api-access-5th2s\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:35.003368 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.003349 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf12e6e-add2-4904-97a1-32e9f5db5af2-openshift-service-ca\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:35.005183 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.005166 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cf12e6e-add2-4904-97a1-32e9f5db5af2-maas-api-tls\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:35.017379 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.017350 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th2s\" (UniqueName: \"kubernetes.io/projected/4cf12e6e-add2-4904-97a1-32e9f5db5af2-kube-api-access-5th2s\") pod \"maas-api-5ddd4945c8-52wgr\" (UID: \"4cf12e6e-add2-4904-97a1-32e9f5db5af2\") " pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:35.117136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.117098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:35.303991 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.303849 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5ddd4945c8-52wgr"] Jul 06 23:52:35.306940 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:52:35.306908 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf12e6e_add2_4904_97a1_32e9f5db5af2.slice/crio-2abf5e4a9e78284611e8be145ab05e1a3bd6ddd775dab6cf18b2e28cd259969e WatchSource:0}: Error finding container 2abf5e4a9e78284611e8be145ab05e1a3bd6ddd775dab6cf18b2e28cd259969e: Status 404 returned error can't find the container with id 2abf5e4a9e78284611e8be145ab05e1a3bd6ddd775dab6cf18b2e28cd259969e Jul 06 23:52:35.557025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.556990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5ddd4945c8-52wgr" event={"ID":"4cf12e6e-add2-4904-97a1-32e9f5db5af2","Type":"ContainerStarted","Data":"2abf5e4a9e78284611e8be145ab05e1a3bd6ddd775dab6cf18b2e28cd259969e"} Jul 06 23:52:35.778133 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.778092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:35.943530 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:35.943445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:36.036723 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.036686 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lw5ck"] Jul 06 23:52:36.039557 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.039537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.047936 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.047910 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 23:52:36.051757 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.051734 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lw5ck"] Jul 06 23:52:36.111459 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.111423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a1830b-dcd2-4e3d-b888-ef32b1f84442-plugins-config-volume\") pod \"payload-processing-58f86c7798-lw5ck\" (UID: \"92a1830b-dcd2-4e3d-b888-ef32b1f84442\") " pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.111659 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.111487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whfht\" (UniqueName: \"kubernetes.io/projected/92a1830b-dcd2-4e3d-b888-ef32b1f84442-kube-api-access-whfht\") pod \"payload-processing-58f86c7798-lw5ck\" (UID: \"92a1830b-dcd2-4e3d-b888-ef32b1f84442\") " pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.212285 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.212188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a1830b-dcd2-4e3d-b888-ef32b1f84442-plugins-config-volume\") pod \"payload-processing-58f86c7798-lw5ck\" (UID: \"92a1830b-dcd2-4e3d-b888-ef32b1f84442\") " pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.212285 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.212274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfht\" (UniqueName: \"kubernetes.io/projected/92a1830b-dcd2-4e3d-b888-ef32b1f84442-kube-api-access-whfht\") pod \"payload-processing-58f86c7798-lw5ck\" (UID: \"92a1830b-dcd2-4e3d-b888-ef32b1f84442\") " pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.213080 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.213052 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a1830b-dcd2-4e3d-b888-ef32b1f84442-plugins-config-volume\") pod \"payload-processing-58f86c7798-lw5ck\" (UID: \"92a1830b-dcd2-4e3d-b888-ef32b1f84442\") " pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.240281 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.240248 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfht\" (UniqueName: \"kubernetes.io/projected/92a1830b-dcd2-4e3d-b888-ef32b1f84442-kube-api-access-whfht\") pod \"payload-processing-58f86c7798-lw5ck\" (UID: \"92a1830b-dcd2-4e3d-b888-ef32b1f84442\") " pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.351025 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.350991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:36.758862 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:52:36.758825 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a1830b_dcd2_4e3d_b888_ef32b1f84442.slice/crio-0fcaa385860ddb1bd99679253961c919d2175aed1323a50b6e859ef524a3989d WatchSource:0}: Error finding container 0fcaa385860ddb1bd99679253961c919d2175aed1323a50b6e859ef524a3989d: Status 404 returned error can't find the container with id 0fcaa385860ddb1bd99679253961c919d2175aed1323a50b6e859ef524a3989d Jul 06 23:52:36.758996 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:36.758951 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lw5ck"] Jul 06 23:52:37.571089 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:37.571039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5ddd4945c8-52wgr" event={"ID":"4cf12e6e-add2-4904-97a1-32e9f5db5af2","Type":"ContainerStarted","Data":"dd5b58082ed139eff31bc7aa21f89c6b377ffe4402a7b178d3a30c07e1d9673a"} Jul 06 23:52:37.571544 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:37.571109 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:37.572497 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:37.572473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" event={"ID":"92a1830b-dcd2-4e3d-b888-ef32b1f84442","Type":"ContainerStarted","Data":"0fcaa385860ddb1bd99679253961c919d2175aed1323a50b6e859ef524a3989d"} Jul 06 23:52:37.608590 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:37.608530 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5ddd4945c8-52wgr" podStartSLOduration=2.293476465 podStartE2EDuration="3.60850979s" totalImagesPullingTime="1.315033325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:52:37.606755633 +0000 UTC m=+587.633660864" watchObservedRunningTime="2026-07-06 23:52:37.60850979 +0000 UTC m=+587.635415021" Jul 06 23:52:38.578416 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:38.578377 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" event={"ID":"92a1830b-dcd2-4e3d-b888-ef32b1f84442","Type":"ContainerStarted","Data":"db99d2106982b8d7c32dbdf4cc4ef5224cfbf7c11e3b93703cc6e6642c0953a6"} Jul 06 23:52:38.578831 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:38.578444 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:38.968209 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:38.968114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:43.583923 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:43.583888 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5ddd4945c8-52wgr" Jul 06 23:52:43.628974 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:43.628918 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" podStartSLOduration=7.6334537860000005 podStartE2EDuration="8.628901019s" totalImagesPullingTime="995.447233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:52:38.626339958 +0000 UTC m=+588.653245187" watchObservedRunningTime="2026-07-06 23:52:43.628901019 +0000 UTC m=+593.655806247" Jul 06 23:52:45.486485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:45.486451 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:45.989299 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:45.989264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:52:49.585473 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:49.585442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-lw5ck" Jul 06 23:52:50.429220 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:50.429187 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:52:50.430333 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:50.430311 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:52:50.432654 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:50.432622 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:52:50.433828 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:50.433812 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:52:55.360138 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:52:55.360106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:53:04.250274 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.250240 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt"] Jul 06 23:53:04.253965 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.253945 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.260296 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.260258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 23:53:04.260296 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.260292 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f2qb9\"" Jul 06 23:53:04.260470 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.260414 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 23:53:04.261190 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.261167 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 23:53:04.292572 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.292533 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt"] Jul 06 23:53:04.333122 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.333092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:53:04.355314 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.355280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca189b38-4f40-4d5e-aaec-f63a5900b29a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.355314 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.355317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4wg\" (UniqueName: \"kubernetes.io/projected/ca189b38-4f40-4d5e-aaec-f63a5900b29a-kube-api-access-jb4wg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.355482 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.355344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.355482 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.355406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.355482 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.355434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.355580 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.355491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456093 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456093 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456379 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456379 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca189b38-4f40-4d5e-aaec-f63a5900b29a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4wg\" (UniqueName: \"kubernetes.io/projected/ca189b38-4f40-4d5e-aaec-f63a5900b29a-kube-api-access-jb4wg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456588 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456523 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456588 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456563 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.456755 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.456733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.458642 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.458614 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca189b38-4f40-4d5e-aaec-f63a5900b29a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.458912 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.458893 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca189b38-4f40-4d5e-aaec-f63a5900b29a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.467540 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.467518 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4wg\" (UniqueName: \"kubernetes.io/projected/ca189b38-4f40-4d5e-aaec-f63a5900b29a-kube-api-access-jb4wg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2fmt\" (UID: \"ca189b38-4f40-4d5e-aaec-f63a5900b29a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.564610 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.564552 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:04.718145 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:04.718105 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt"] Jul 06 23:53:04.720313 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:53:04.720286 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca189b38_4f40_4d5e_aaec_f63a5900b29a.slice/crio-8a57e7be780c2573f851e5dcecf0df31f4dcac2b89108a41c9945bf09be0d72f WatchSource:0}: Error finding container 8a57e7be780c2573f851e5dcecf0df31f4dcac2b89108a41c9945bf09be0d72f: Status 404 returned error can't find the container with id 8a57e7be780c2573f851e5dcecf0df31f4dcac2b89108a41c9945bf09be0d72f Jul 06 23:53:05.709704 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:05.709661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" event={"ID":"ca189b38-4f40-4d5e-aaec-f63a5900b29a","Type":"ContainerStarted","Data":"8a57e7be780c2573f851e5dcecf0df31f4dcac2b89108a41c9945bf09be0d72f"} Jul 06 23:53:06.715303 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:06.715266 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" event={"ID":"ca189b38-4f40-4d5e-aaec-f63a5900b29a","Type":"ContainerStarted","Data":"8f23f847f5ee0cc0d21fdda4ffc964c66eb0eaa8edb1490b3a1088bfaa833032"} Jul 06 23:53:06.741964 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:06.741916 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" podStartSLOduration=1.620468145 podStartE2EDuration="2.741901761s" totalImagesPullingTime="1.121433616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:53:06.741369225 +0000 UTC m=+616.768274451" watchObservedRunningTime="2026-07-06 23:53:06.741901761 +0000 UTC m=+616.768806989" Jul 06 23:53:14.565485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:14.565387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:14.565485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:14.565455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:14.578010 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:14.577980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:14.764286 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:14.764256 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2fmt" Jul 06 23:53:15.700948 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:15.700911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:53:35.528519 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.528485 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x"] Jul 06 23:53:35.532828 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.532808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.535727 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.535678 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 23:53:35.544437 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.544413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x"] Jul 06 23:53:35.564163 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.564128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:53:35.643842 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.643801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.643842 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.643849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.644081 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.643881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.644081 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.643898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.644081 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.644024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th9q8\" (UniqueName: \"kubernetes.io/projected/b4002879-478f-4b5d-8a40-fc324acc7e3d-kube-api-access-th9q8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.644227 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.644080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4002879-478f-4b5d-8a40-fc324acc7e3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.744572 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.744533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.744791 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.744595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.744791 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.744716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.744932 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.744809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th9q8\" (UniqueName: \"kubernetes.io/projected/b4002879-478f-4b5d-8a40-fc324acc7e3d-kube-api-access-th9q8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.744932 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.744848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4002879-478f-4b5d-8a40-fc324acc7e3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.744932 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.744905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.745137 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.745078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.745254 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.745230 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.745376 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.745259 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.746898 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.746876 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4002879-478f-4b5d-8a40-fc324acc7e3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.747290 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.747269 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4002879-478f-4b5d-8a40-fc324acc7e3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.752846 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.752826 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th9q8\" (UniqueName: \"kubernetes.io/projected/b4002879-478f-4b5d-8a40-fc324acc7e3d-kube-api-access-th9q8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x\" (UID: \"b4002879-478f-4b5d-8a40-fc324acc7e3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.843234 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.843133 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:35.983213 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.983068 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x"] Jul 06 23:53:35.986164 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:53:35.986135 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4002879_478f_4b5d_8a40_fc324acc7e3d.slice/crio-3de646d62cf1bc69881acc731432ed5afa51445543da4d392034bcd07dcae1d1 WatchSource:0}: Error finding container 3de646d62cf1bc69881acc731432ed5afa51445543da4d392034bcd07dcae1d1: Status 404 returned error can't find the container with id 3de646d62cf1bc69881acc731432ed5afa51445543da4d392034bcd07dcae1d1 Jul 06 23:53:35.987426 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:35.987402 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:53:36.864260 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:36.864222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" event={"ID":"b4002879-478f-4b5d-8a40-fc324acc7e3d","Type":"ContainerStarted","Data":"340234ef257fdc58f71330b2f7a37f3e76cd0cfcfdd8ef1384b8e08bf6061383"} Jul 06 23:53:36.864680 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:36.864272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" event={"ID":"b4002879-478f-4b5d-8a40-fc324acc7e3d","Type":"ContainerStarted","Data":"3de646d62cf1bc69881acc731432ed5afa51445543da4d392034bcd07dcae1d1"} Jul 06 23:53:36.882014 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:36.881966 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" podStartSLOduration=1.6946317579999999 podStartE2EDuration="1.881950815s" totalImagesPullingTime="187.319057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:53:36.880600601 +0000 UTC m=+646.907505830" watchObservedRunningTime="2026-07-06 23:53:36.881950815 +0000 UTC m=+646.908856043" Jul 06 23:53:45.843280 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:45.843242 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:45.843829 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:45.843313 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:45.855965 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:45.855936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:45.923374 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:45.923344 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x" Jul 06 23:53:48.256009 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:48.255964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:53:55.849328 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:53:55.849286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:05.632825 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:05.632785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:21.856621 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:21.856583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:21.957618 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:21.957586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:29.808418 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:29.808379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:45.752661 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:45.752552 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cd5bf955f-775lp"] Jul 06 23:54:45.753063 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:45.752805 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-cd5bf955f-775lp" podUID="c9dd945f-342b-469a-b26d-8706c457b160" containerName="authorino" containerID="cri-o://331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3" gracePeriod=30 Jul 06 23:54:46.020748 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.020722 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:54:46.059073 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.059030 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfkvm\" (UniqueName: \"kubernetes.io/projected/c9dd945f-342b-469a-b26d-8706c457b160-kube-api-access-pfkvm\") pod \"c9dd945f-342b-469a-b26d-8706c457b160\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " Jul 06 23:54:46.059073 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.059080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9dd945f-342b-469a-b26d-8706c457b160-tls-cert\") pod \"c9dd945f-342b-469a-b26d-8706c457b160\" (UID: \"c9dd945f-342b-469a-b26d-8706c457b160\") " Jul 06 23:54:46.061221 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.061186 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9dd945f-342b-469a-b26d-8706c457b160-kube-api-access-pfkvm" (OuterVolumeSpecName: "kube-api-access-pfkvm") pod "c9dd945f-342b-469a-b26d-8706c457b160" (UID: "c9dd945f-342b-469a-b26d-8706c457b160"). InnerVolumeSpecName "kube-api-access-pfkvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:54:46.069288 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.069264 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9dd945f-342b-469a-b26d-8706c457b160-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c9dd945f-342b-469a-b26d-8706c457b160" (UID: "c9dd945f-342b-469a-b26d-8706c457b160"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:54:46.160157 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.160114 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkvm\" (UniqueName: \"kubernetes.io/projected/c9dd945f-342b-469a-b26d-8706c457b160-kube-api-access-pfkvm\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:54:46.160157 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.160147 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9dd945f-342b-469a-b26d-8706c457b160-tls-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:54:46.208413 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.208381 2564 generic.go:356] "Generic (PLEG): container finished" podID="c9dd945f-342b-469a-b26d-8706c457b160" containerID="331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3" exitCode=0 Jul 06 23:54:46.208582 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.208420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd5bf955f-775lp" event={"ID":"c9dd945f-342b-469a-b26d-8706c457b160","Type":"ContainerDied","Data":"331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3"} Jul 06 23:54:46.208582 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.208429 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd5bf955f-775lp" Jul 06 23:54:46.208582 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.208453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd5bf955f-775lp" event={"ID":"c9dd945f-342b-469a-b26d-8706c457b160","Type":"ContainerDied","Data":"823984bff7c8fe816ab96071998759bcd47730688733291993343bc8056fbabb"} Jul 06 23:54:46.208582 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.208470 2564 scope.go:117] "RemoveContainer" containerID="331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3" Jul 06 23:54:46.218144 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.218127 2564 scope.go:117] "RemoveContainer" containerID="331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3" Jul 06 23:54:46.218413 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:54:46.218393 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3\": container with ID starting with 331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3 not found: ID does not exist" containerID="331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3" Jul 06 23:54:46.218454 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.218422 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3"} err="failed to get container status \"331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3\": rpc error: code = NotFound desc = could not find container \"331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3\": container with ID starting with 331ea1047053b76b8a18201b161bb0a49ed74e5675e3f76fbc923b1c4acc3bf3 not found: ID does not exist" Jul 06 23:54:46.235562 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.235530 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cd5bf955f-775lp"] Jul 06 23:54:46.241865 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.241834 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cd5bf955f-775lp"] Jul 06 23:54:46.263221 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.263125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:46.359432 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.359394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:46.659434 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:46.659402 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9dd945f-342b-469a-b26d-8706c457b160" path="/var/lib/kubelet/pods/c9dd945f-342b-469a-b26d-8706c457b160/volumes" Jul 06 23:54:47.585753 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:47.585715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:54:47.687523 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:54:47.687479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:55:24.296960 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:24.296926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:55:58.604668 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:58.604619 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:55:58.708492 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:58.708451 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-jbrsr"] Jul 06 23:55:58.708682 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:58.708658 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" podUID="e16cb968-e8b3-4100-b784-dfa8fbad9722" containerName="manager" containerID="cri-o://46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041" gracePeriod=30 Jul 06 23:55:58.963224 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:58.963198 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:55:59.107577 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.107534 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npmfx\" (UniqueName: \"kubernetes.io/projected/e16cb968-e8b3-4100-b784-dfa8fbad9722-kube-api-access-npmfx\") pod \"e16cb968-e8b3-4100-b784-dfa8fbad9722\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " Jul 06 23:55:59.107807 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.107661 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert\") pod \"e16cb968-e8b3-4100-b784-dfa8fbad9722\" (UID: \"e16cb968-e8b3-4100-b784-dfa8fbad9722\") " Jul 06 23:55:59.109677 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.109646 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e16cb968-e8b3-4100-b784-dfa8fbad9722" (UID: "e16cb968-e8b3-4100-b784-dfa8fbad9722"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:55:59.109816 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.109733 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16cb968-e8b3-4100-b784-dfa8fbad9722-kube-api-access-npmfx" (OuterVolumeSpecName: "kube-api-access-npmfx") pod "e16cb968-e8b3-4100-b784-dfa8fbad9722" (UID: "e16cb968-e8b3-4100-b784-dfa8fbad9722"). InnerVolumeSpecName "kube-api-access-npmfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:55:59.208786 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.208710 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmfx\" (UniqueName: \"kubernetes.io/projected/e16cb968-e8b3-4100-b784-dfa8fbad9722-kube-api-access-npmfx\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:55:59.208786 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.208736 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e16cb968-e8b3-4100-b784-dfa8fbad9722-webhook-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 06 23:55:59.558726 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.558694 2564 generic.go:356] "Generic (PLEG): container finished" podID="e16cb968-e8b3-4100-b784-dfa8fbad9722" containerID="46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041" exitCode=0 Jul 06 23:55:59.558895 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.558742 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" Jul 06 23:55:59.558895 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.558785 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" event={"ID":"e16cb968-e8b3-4100-b784-dfa8fbad9722","Type":"ContainerDied","Data":"46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041"} Jul 06 23:55:59.558895 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.558843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696cfbbf9b-jbrsr" event={"ID":"e16cb968-e8b3-4100-b784-dfa8fbad9722","Type":"ContainerDied","Data":"5a781e358fc1a4ef73d881402a6ccbcfe8981c51b08d79d498524779bd219db9"} Jul 06 23:55:59.558895 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.558873 2564 scope.go:117] "RemoveContainer" containerID="46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041" Jul 06 23:55:59.567785 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.567753 2564 scope.go:117] "RemoveContainer" containerID="46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041" Jul 06 23:55:59.568027 ip-10-0-132-183 kubenswrapper[2564]: E0706 23:55:59.568009 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041\": container with ID starting with 46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041 not found: ID does not exist" containerID="46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041" Jul 06 23:55:59.568077 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.568035 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041"} err="failed to get container status \"46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041\": rpc error: code = NotFound desc = could not find container \"46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041\": container with ID starting with 46abf297110de705c66f1eb0db783fff19cd828f3de9238e05c892a1eb032041 not found: ID does not exist" Jul 06 23:55:59.579648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.579609 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-jbrsr"] Jul 06 23:55:59.584213 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.584192 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-jbrsr"] Jul 06 23:55:59.964207 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:55:59.964120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:00.258532 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.258443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:00.375436 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.375400 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-pm4gt"] Jul 06 23:56:00.375860 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.375842 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9dd945f-342b-469a-b26d-8706c457b160" containerName="authorino" Jul 06 23:56:00.375978 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.375862 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9dd945f-342b-469a-b26d-8706c457b160" containerName="authorino" Jul 06 23:56:00.375978 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.375875 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16cb968-e8b3-4100-b784-dfa8fbad9722" containerName="manager" Jul 06 23:56:00.375978 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.375883 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16cb968-e8b3-4100-b784-dfa8fbad9722" containerName="manager" Jul 06 23:56:00.376136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.375988 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9dd945f-342b-469a-b26d-8706c457b160" containerName="authorino" Jul 06 23:56:00.376136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.376001 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16cb968-e8b3-4100-b784-dfa8fbad9722" containerName="manager" Jul 06 23:56:00.379979 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.379957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.383053 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.383032 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 23:56:00.383131 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.383037 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 23:56:00.383131 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.383033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddfvt\"" Jul 06 23:56:00.390460 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.390440 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-pm4gt"] Jul 06 23:56:00.521304 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.521221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eaddac64-3801-478c-9737-f59ba2f8c621-webhook-cert\") pod \"maas-controller-696cfbbf9b-pm4gt\" (UID: \"eaddac64-3801-478c-9737-f59ba2f8c621\") " pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.521304 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.521275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk5c6\" (UniqueName: \"kubernetes.io/projected/eaddac64-3801-478c-9737-f59ba2f8c621-kube-api-access-rk5c6\") pod \"maas-controller-696cfbbf9b-pm4gt\" (UID: \"eaddac64-3801-478c-9737-f59ba2f8c621\") " pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.622485 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.622456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eaddac64-3801-478c-9737-f59ba2f8c621-webhook-cert\") pod \"maas-controller-696cfbbf9b-pm4gt\" (UID: \"eaddac64-3801-478c-9737-f59ba2f8c621\") " pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.622676 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.622516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5c6\" (UniqueName: \"kubernetes.io/projected/eaddac64-3801-478c-9737-f59ba2f8c621-kube-api-access-rk5c6\") pod \"maas-controller-696cfbbf9b-pm4gt\" (UID: \"eaddac64-3801-478c-9737-f59ba2f8c621\") " pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.624888 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.624865 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eaddac64-3801-478c-9737-f59ba2f8c621-webhook-cert\") pod \"maas-controller-696cfbbf9b-pm4gt\" (UID: \"eaddac64-3801-478c-9737-f59ba2f8c621\") " pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.631370 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.631342 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5c6\" (UniqueName: \"kubernetes.io/projected/eaddac64-3801-478c-9737-f59ba2f8c621-kube-api-access-rk5c6\") pod \"maas-controller-696cfbbf9b-pm4gt\" (UID: \"eaddac64-3801-478c-9737-f59ba2f8c621\") " pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.654946 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.654924 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16cb968-e8b3-4100-b784-dfa8fbad9722" path="/var/lib/kubelet/pods/e16cb968-e8b3-4100-b784-dfa8fbad9722/volumes" Jul 06 23:56:00.690848 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.690825 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:00.825965 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:00.825941 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-696cfbbf9b-pm4gt"] Jul 06 23:56:00.829017 ip-10-0-132-183 kubenswrapper[2564]: W0706 23:56:00.828972 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaddac64_3801_478c_9737_f59ba2f8c621.slice/crio-9c6ff67ea0277882165381f6e2df2876d5fc59c1328d498fdfdfdb32eea488f3 WatchSource:0}: Error finding container 9c6ff67ea0277882165381f6e2df2876d5fc59c1328d498fdfdfdb32eea488f3: Status 404 returned error can't find the container with id 9c6ff67ea0277882165381f6e2df2876d5fc59c1328d498fdfdfdb32eea488f3 Jul 06 23:56:01.569317 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:01.569273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" event={"ID":"eaddac64-3801-478c-9737-f59ba2f8c621","Type":"ContainerStarted","Data":"869c4135daa0e379b25930200c1385016961e1dffb7d75d23cc2426be5cb628c"} Jul 06 23:56:01.569803 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:01.569445 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:01.569803 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:01.569468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" event={"ID":"eaddac64-3801-478c-9737-f59ba2f8c621","Type":"ContainerStarted","Data":"9c6ff67ea0277882165381f6e2df2876d5fc59c1328d498fdfdfdb32eea488f3"} Jul 06 23:56:01.589047 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:01.588971 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" podStartSLOduration=1.258979652 podStartE2EDuration="1.588955201s" totalImagesPullingTime="329.975549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:01.587652333 +0000 UTC m=+791.614557554" watchObservedRunningTime="2026-07-06 23:56:01.588955201 +0000 UTC m=+791.615860430" Jul 06 23:56:12.580442 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:12.580402 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-696cfbbf9b-pm4gt" Jul 06 23:56:40.922136 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:40.922099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:41.014312 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:41.014275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:41.118264 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:41.118223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:46.917604 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:46.917564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:46.984705 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:46.984674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:47.090648 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:47.090595 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:47.189590 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:47.189507 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:47.304506 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:47.304472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.263147 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:53.263106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.364643 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:53.364596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.467787 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:56:53.467756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:50.466884 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:57:50.466813 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:57:50.469668 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:57:50.469623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 06 23:57:50.471102 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:57:50.471081 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:57:50.473287 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:57:50.473269 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 06 23:59:49.757419 ip-10-0-132-183 kubenswrapper[2564]: I0706 23:59:49.757331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:00.160051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.160013 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723040-94xpf"] Jul 07 00:00:00.163453 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.163432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" Jul 07 00:00:00.169556 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.169536 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:00:00.193603 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.193569 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723040-94xpf"] Jul 07 00:00:00.241885 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.241853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6ng\" (UniqueName: \"kubernetes.io/projected/c971ef3f-1281-4d73-a14f-cdd896dff546-kube-api-access-rr6ng\") pod \"maas-api-key-cleanup-29723040-94xpf\" (UID: \"c971ef3f-1281-4d73-a14f-cdd896dff546\") " pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" Jul 07 00:00:00.259264 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.259232 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29723040-7vx28"] Jul 07 00:00:00.262841 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.262826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.265269 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.265242 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Jul 07 00:00:00.265408 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.265245 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-29grb\"" Jul 07 00:00:00.271268 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.271241 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29723040-7vx28"] Jul 07 00:00:00.342784 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.342749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6ng\" (UniqueName: \"kubernetes.io/projected/c971ef3f-1281-4d73-a14f-cdd896dff546-kube-api-access-rr6ng\") pod \"maas-api-key-cleanup-29723040-94xpf\" (UID: \"c971ef3f-1281-4d73-a14f-cdd896dff546\") " pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" Jul 07 00:00:00.342956 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.342817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9329f926-1025-4c26-bbb9-bbe6ef08a871-serviceca\") pod \"image-pruner-29723040-7vx28\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.342956 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.342874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6qp\" (UniqueName: \"kubernetes.io/projected/9329f926-1025-4c26-bbb9-bbe6ef08a871-kube-api-access-rh6qp\") pod \"image-pruner-29723040-7vx28\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.351259 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.351229 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6ng\" (UniqueName: \"kubernetes.io/projected/c971ef3f-1281-4d73-a14f-cdd896dff546-kube-api-access-rr6ng\") pod \"maas-api-key-cleanup-29723040-94xpf\" (UID: \"c971ef3f-1281-4d73-a14f-cdd896dff546\") " pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" Jul 07 00:00:00.443428 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.443341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9329f926-1025-4c26-bbb9-bbe6ef08a871-serviceca\") pod \"image-pruner-29723040-7vx28\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.443428 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.443409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6qp\" (UniqueName: \"kubernetes.io/projected/9329f926-1025-4c26-bbb9-bbe6ef08a871-kube-api-access-rh6qp\") pod \"image-pruner-29723040-7vx28\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.444051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.444033 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9329f926-1025-4c26-bbb9-bbe6ef08a871-serviceca\") pod \"image-pruner-29723040-7vx28\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.452567 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.452547 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6qp\" (UniqueName: \"kubernetes.io/projected/9329f926-1025-4c26-bbb9-bbe6ef08a871-kube-api-access-rh6qp\") pod \"image-pruner-29723040-7vx28\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.508544 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.508516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" Jul 07 00:00:00.573764 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.573729 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:00.714557 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.714525 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29723040-7vx28"] Jul 07 00:00:00.717465 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:00:00.717441 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9329f926_1025_4c26_bbb9_bbe6ef08a871.slice/crio-180398edbd5e9db1d6bb2cd9a563ee61ef0c8ac4c371f2ced7213d6dd4bda621 WatchSource:0}: Error finding container 180398edbd5e9db1d6bb2cd9a563ee61ef0c8ac4c371f2ced7213d6dd4bda621: Status 404 returned error can't find the container with id 180398edbd5e9db1d6bb2cd9a563ee61ef0c8ac4c371f2ced7213d6dd4bda621 Jul 07 00:00:00.863468 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.863441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723040-94xpf"] Jul 07 00:00:00.866134 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:00:00.866110 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc971ef3f_1281_4d73_a14f_cdd896dff546.slice/crio-7d214ce6c2ef292d65d6e588a0b82bab37fee68af38544d367f5d186de9c72ba WatchSource:0}: Error finding container 7d214ce6c2ef292d65d6e588a0b82bab37fee68af38544d367f5d186de9c72ba: Status 404 returned error can't find the container with id 7d214ce6c2ef292d65d6e588a0b82bab37fee68af38544d367f5d186de9c72ba Jul 07 00:00:00.868103 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:00.868087 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:00:01.722852 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:01.722812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" event={"ID":"c971ef3f-1281-4d73-a14f-cdd896dff546","Type":"ContainerStarted","Data":"7d214ce6c2ef292d65d6e588a0b82bab37fee68af38544d367f5d186de9c72ba"} Jul 07 00:00:01.724210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:01.724181 2564 generic.go:356] "Generic (PLEG): container finished" podID="9329f926-1025-4c26-bbb9-bbe6ef08a871" containerID="07ec4ea0bbe7679e9df24f07f2bfdb2b234730105e8e6c090a828dfb89d79529" exitCode=0 Jul 07 00:00:01.724303 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:01.724269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-7vx28" event={"ID":"9329f926-1025-4c26-bbb9-bbe6ef08a871","Type":"ContainerDied","Data":"07ec4ea0bbe7679e9df24f07f2bfdb2b234730105e8e6c090a828dfb89d79529"} Jul 07 00:00:01.724344 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:01.724312 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-7vx28" event={"ID":"9329f926-1025-4c26-bbb9-bbe6ef08a871","Type":"ContainerStarted","Data":"180398edbd5e9db1d6bb2cd9a563ee61ef0c8ac4c371f2ced7213d6dd4bda621"} Jul 07 00:00:02.871735 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:02.871711 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:02.971409 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:02.971371 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh6qp\" (UniqueName: \"kubernetes.io/projected/9329f926-1025-4c26-bbb9-bbe6ef08a871-kube-api-access-rh6qp\") pod \"9329f926-1025-4c26-bbb9-bbe6ef08a871\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " Jul 07 00:00:02.971605 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:02.971465 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9329f926-1025-4c26-bbb9-bbe6ef08a871-serviceca\") pod \"9329f926-1025-4c26-bbb9-bbe6ef08a871\" (UID: \"9329f926-1025-4c26-bbb9-bbe6ef08a871\") " Jul 07 00:00:02.971872 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:02.971837 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9329f926-1025-4c26-bbb9-bbe6ef08a871-serviceca" (OuterVolumeSpecName: "serviceca") pod "9329f926-1025-4c26-bbb9-bbe6ef08a871" (UID: "9329f926-1025-4c26-bbb9-bbe6ef08a871"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:00:02.973581 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:02.973555 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9329f926-1025-4c26-bbb9-bbe6ef08a871-kube-api-access-rh6qp" (OuterVolumeSpecName: "kube-api-access-rh6qp") pod "9329f926-1025-4c26-bbb9-bbe6ef08a871" (UID: "9329f926-1025-4c26-bbb9-bbe6ef08a871"). InnerVolumeSpecName "kube-api-access-rh6qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:03.072358 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:03.072303 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6qp\" (UniqueName: \"kubernetes.io/projected/9329f926-1025-4c26-bbb9-bbe6ef08a871-kube-api-access-rh6qp\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:00:03.072358 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:03.072354 2564 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9329f926-1025-4c26-bbb9-bbe6ef08a871-serviceca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:00:03.736048 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:03.736018 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-7vx28" Jul 07 00:00:03.736227 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:03.736058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-7vx28" event={"ID":"9329f926-1025-4c26-bbb9-bbe6ef08a871","Type":"ContainerDied","Data":"180398edbd5e9db1d6bb2cd9a563ee61ef0c8ac4c371f2ced7213d6dd4bda621"} Jul 07 00:00:03.736227 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:03.736107 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="180398edbd5e9db1d6bb2cd9a563ee61ef0c8ac4c371f2ced7213d6dd4bda621" Jul 07 00:00:12.319007 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:12.318968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:28.861334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:28.861284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" event={"ID":"c971ef3f-1281-4d73-a14f-cdd896dff546","Type":"ContainerStarted","Data":"83499b43e676f5cbaf48fab60d5580c6f636393b53a430333a83df2a2b4e1991"} Jul 07 00:00:28.879018 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:28.878964 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" podStartSLOduration=1.727912725 podStartE2EDuration="28.878946086s" totalImagesPullingTime="27.151033361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:28.87705025 +0000 UTC m=+1058.903955543" watchObservedRunningTime="2026-07-07 00:00:28.878946086 +0000 UTC m=+1058.905851315" Jul 07 00:00:29.869243 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:29.869204 2564 generic.go:356] "Generic (PLEG): container finished" podID="c971ef3f-1281-4d73-a14f-cdd896dff546" containerID="83499b43e676f5cbaf48fab60d5580c6f636393b53a430333a83df2a2b4e1991" exitCode=0 Jul 07 00:00:29.869656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:29.869289 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" event={"ID":"c971ef3f-1281-4d73-a14f-cdd896dff546","Type":"ContainerDied","Data":"83499b43e676f5cbaf48fab60d5580c6f636393b53a430333a83df2a2b4e1991"} Jul 07 00:00:31.007668 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:31.007612 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" Jul 07 00:00:31.128988 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:31.128878 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr6ng\" (UniqueName: \"kubernetes.io/projected/c971ef3f-1281-4d73-a14f-cdd896dff546-kube-api-access-rr6ng\") pod \"c971ef3f-1281-4d73-a14f-cdd896dff546\" (UID: \"c971ef3f-1281-4d73-a14f-cdd896dff546\") " Jul 07 00:00:31.131017 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:31.130985 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c971ef3f-1281-4d73-a14f-cdd896dff546-kube-api-access-rr6ng" (OuterVolumeSpecName: "kube-api-access-rr6ng") pod "c971ef3f-1281-4d73-a14f-cdd896dff546" (UID: "c971ef3f-1281-4d73-a14f-cdd896dff546"). InnerVolumeSpecName "kube-api-access-rr6ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:31.229943 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:31.229903 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr6ng\" (UniqueName: \"kubernetes.io/projected/c971ef3f-1281-4d73-a14f-cdd896dff546-kube-api-access-rr6ng\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:00:31.880086 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:31.880058 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" Jul 07 00:00:31.880262 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:31.880055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723040-94xpf" event={"ID":"c971ef3f-1281-4d73-a14f-cdd896dff546","Type":"ContainerDied","Data":"7d214ce6c2ef292d65d6e588a0b82bab37fee68af38544d367f5d186de9c72ba"} Jul 07 00:00:31.880262 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:00:31.880182 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d214ce6c2ef292d65d6e588a0b82bab37fee68af38544d367f5d186de9c72ba" Jul 07 00:01:43.872493 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:01:43.872457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:49.506082 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:01:49.506044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:50.497299 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:02:50.497218 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:02:50.498334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:02:50.498316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:02:50.501216 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:02:50.501195 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:02:50.501953 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:02:50.501934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:06:55.471148 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:06:55.471074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:06:56.789855 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:06:56.789816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:29.400312 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:29.400274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:39.803599 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:39.803564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:48.722807 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:48.722770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:50.534537 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:50.534508 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:07:50.535919 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:50.535894 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:07:50.538529 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:50.538510 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:07:50.539834 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:50.539813 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:07:53.011999 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:53.011965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:56.027476 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:56.027429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:56.121336 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:56.121302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:56.488405 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:07:56.488309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:02.475487 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:02.475452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:02.576482 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:02.576441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:02.718078 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:02.718040 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:02.929743 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:02.929712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:03.922640 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.922597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:03.977378 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.977347 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4"] Jul 07 00:08:03.977752 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.977737 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9329f926-1025-4c26-bbb9-bbe6ef08a871" containerName="image-pruner" Jul 07 00:08:03.977814 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.977753 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9329f926-1025-4c26-bbb9-bbe6ef08a871" containerName="image-pruner" Jul 07 00:08:03.977814 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.977777 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c971ef3f-1281-4d73-a14f-cdd896dff546" containerName="cleanup" Jul 07 00:08:03.977814 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.977782 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c971ef3f-1281-4d73-a14f-cdd896dff546" containerName="cleanup" Jul 07 00:08:03.977927 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.977850 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9329f926-1025-4c26-bbb9-bbe6ef08a871" containerName="image-pruner" Jul 07 00:08:03.977927 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.977860 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c971ef3f-1281-4d73-a14f-cdd896dff546" containerName="cleanup" Jul 07 00:08:03.981043 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.981023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:03.985715 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.985695 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-27546a9a-serving-cert\"" Jul 07 00:08:03.985715 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.985708 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:08:03.998266 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:03.998242 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4"] Jul 07 00:08:04.008062 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.008023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:04.143929 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.143891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-openshift-service-ca\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.144105 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.143950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-maas-api-tls\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.144105 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.144037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8nmg\" (UniqueName: \"kubernetes.io/projected/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-kube-api-access-w8nmg\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.146554 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.146515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:04.245077 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.244979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-openshift-service-ca\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.245077 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.245067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-maas-api-tls\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.245271 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.245096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8nmg\" (UniqueName: \"kubernetes.io/projected/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-kube-api-access-w8nmg\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.245680 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.245659 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-openshift-service-ca\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.247590 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.247565 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-maas-api-tls\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.254808 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.254775 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8nmg\" (UniqueName: \"kubernetes.io/projected/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-kube-api-access-w8nmg\") pod \"maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.293712 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.293677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:04.300796 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.300768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:04.433581 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.433556 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4"] Jul 07 00:08:04.435794 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:08:04.435753 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e10d6c2_7c65_49f4_ba5e_d39f9015a477.slice/crio-4e64bb41c691e2575a8ebd9de580588165dc11e8520ef670f17c07378dacbdb3 WatchSource:0}: Error finding container 4e64bb41c691e2575a8ebd9de580588165dc11e8520ef670f17c07378dacbdb3: Status 404 returned error can't find the container with id 4e64bb41c691e2575a8ebd9de580588165dc11e8520ef670f17c07378dacbdb3 Jul 07 00:08:04.437180 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.437157 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:08:04.457670 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.457622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:04.561924 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:04.561891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:05.061755 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:05.061718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" event={"ID":"1e10d6c2-7c65-49f4-ba5e-d39f9015a477","Type":"ContainerStarted","Data":"4e64bb41c691e2575a8ebd9de580588165dc11e8520ef670f17c07378dacbdb3"} Jul 07 00:08:06.067548 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:06.067510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" event={"ID":"1e10d6c2-7c65-49f4-ba5e-d39f9015a477","Type":"ContainerStarted","Data":"692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95"} Jul 07 00:08:06.068005 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:06.067599 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:06.088641 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:06.088575 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" podStartSLOduration=2.299825212 podStartE2EDuration="3.088555025s" totalImagesPullingTime="788.729813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:08:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:08:06.087491186 +0000 UTC m=+1516.114396415" watchObservedRunningTime="2026-07-07 00:08:06.088555025 +0000 UTC m=+1516.115460274" Jul 07 00:08:06.389074 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:06.389042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:08.985134 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:08.985102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:09.202254 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.202220 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:09.304660 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.304611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:09.409525 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.409484 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4"] Jul 07 00:08:09.409840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.409807 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" podUID="1e10d6c2-7c65-49f4-ba5e-d39f9015a477" containerName="maas-api" containerID="cri-o://692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95" gracePeriod=30 Jul 07 00:08:09.440801 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.440774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:09.664987 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.664962 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:09.798251 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.798209 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-maas-api-tls\") pod \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " Jul 07 00:08:09.798464 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.798351 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-openshift-service-ca\") pod \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " Jul 07 00:08:09.798535 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.798489 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8nmg\" (UniqueName: \"kubernetes.io/projected/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-kube-api-access-w8nmg\") pod \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\" (UID: \"1e10d6c2-7c65-49f4-ba5e-d39f9015a477\") " Jul 07 00:08:09.798839 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.798815 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e10d6c2-7c65-49f4-ba5e-d39f9015a477" (UID: "1e10d6c2-7c65-49f4-ba5e-d39f9015a477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:08:09.800384 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.800362 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e10d6c2-7c65-49f4-ba5e-d39f9015a477" (UID: "1e10d6c2-7c65-49f4-ba5e-d39f9015a477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:08:09.800504 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.800484 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-kube-api-access-w8nmg" (OuterVolumeSpecName: "kube-api-access-w8nmg") pod "1e10d6c2-7c65-49f4-ba5e-d39f9015a477" (UID: "1e10d6c2-7c65-49f4-ba5e-d39f9015a477"). InnerVolumeSpecName "kube-api-access-w8nmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:09.899385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.899302 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:09.899385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.899332 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8nmg\" (UniqueName: \"kubernetes.io/projected/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-kube-api-access-w8nmg\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:09.899385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:09.899343 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e10d6c2-7c65-49f4-ba5e-d39f9015a477-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:10.087715 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.087677 2564 generic.go:356] "Generic (PLEG): container finished" podID="1e10d6c2-7c65-49f4-ba5e-d39f9015a477" containerID="692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95" exitCode=0 Jul 07 00:08:10.088141 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.087728 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" Jul 07 00:08:10.088141 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.087748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" event={"ID":"1e10d6c2-7c65-49f4-ba5e-d39f9015a477","Type":"ContainerDied","Data":"692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95"} Jul 07 00:08:10.088141 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.087802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4" event={"ID":"1e10d6c2-7c65-49f4-ba5e-d39f9015a477","Type":"ContainerDied","Data":"4e64bb41c691e2575a8ebd9de580588165dc11e8520ef670f17c07378dacbdb3"} Jul 07 00:08:10.088141 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.087825 2564 scope.go:117] "RemoveContainer" containerID="692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95" Jul 07 00:08:10.096936 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.096919 2564 scope.go:117] "RemoveContainer" containerID="692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95" Jul 07 00:08:10.097191 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:08:10.097174 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95\": container with ID starting with 692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95 not found: ID does not exist" containerID="692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95" Jul 07 00:08:10.097248 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.097201 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95"} err="failed to get container status \"692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95\": rpc error: code = NotFound desc = could not find container \"692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95\": container with ID starting with 692728fa1d67d4baa34bea494bf4522a2a0a682c16e7a0ac1b826502988b3f95 not found: ID does not exist" Jul 07 00:08:10.121785 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.121758 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4"] Jul 07 00:08:10.123569 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.123550 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-27546a9a-dc8865f6d-zs9t4"] Jul 07 00:08:10.655286 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.655250 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e10d6c2-7c65-49f4-ba5e-d39f9015a477" path="/var/lib/kubelet/pods/1e10d6c2-7c65-49f4-ba5e-d39f9015a477/volumes" Jul 07 00:08:10.704998 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:10.704969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:15.127024 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.126939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:15.229572 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.229536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:15.272851 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.272819 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44"] Jul 07 00:08:15.273196 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.273184 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e10d6c2-7c65-49f4-ba5e-d39f9015a477" containerName="maas-api" Jul 07 00:08:15.273245 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.273198 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e10d6c2-7c65-49f4-ba5e-d39f9015a477" containerName="maas-api" Jul 07 00:08:15.273283 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.273274 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e10d6c2-7c65-49f4-ba5e-d39f9015a477" containerName="maas-api" Jul 07 00:08:15.277963 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.277939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.280372 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.280346 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-76a59d65-openshift-default-dockercfg-rrlfp\"" Jul 07 00:08:15.280493 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.280352 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:08:15.294305 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.294278 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44"] Jul 07 00:08:15.318882 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.318845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:15.390870 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.390784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:15.445418 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-token\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445578 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-certs\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445578 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcfq\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-kube-api-access-4gcfq\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445578 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-data\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445578 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-podinfo\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445773 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-socket\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445773 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-envoy\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445773 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab62ee5-f566-4887-9974-d5ebb66db81c-istiod-ca-cert\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.445773 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.445735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-credential-socket\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546424 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-token\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546424 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-certs\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546665 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcfq\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-kube-api-access-4gcfq\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546665 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-data\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546665 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-podinfo\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546665 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-socket\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546854 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-envoy\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546854 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab62ee5-f566-4887-9974-d5ebb66db81c-istiod-ca-cert\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.546854 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-credential-socket\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.547014 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546884 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-certs\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.547014 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.546968 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-data\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.547109 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.547043 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-socket\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.547152 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.547133 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-credential-socket\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.547454 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.547436 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab62ee5-f566-4887-9974-d5ebb66db81c-istiod-ca-cert\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.548987 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.548965 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-envoy\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.549054 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.549026 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-podinfo\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.554322 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.554297 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-token\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.554398 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.554365 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcfq\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-kube-api-access-4gcfq\") pod \"e2e-ait-76a59d65-openshift-default-76b989965c-ndl44\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.590322 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.590286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:15.730416 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:15.730315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44"] Jul 07 00:08:15.732842 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:08:15.732800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab62ee5_f566_4887_9974_d5ebb66db81c.slice/crio-1f898f4c57d40cdcac82dd3520c5251795add9ae6d9ded21f7d958bd4ea663e9 WatchSource:0}: Error finding container 1f898f4c57d40cdcac82dd3520c5251795add9ae6d9ded21f7d958bd4ea663e9: Status 404 returned error can't find the container with id 1f898f4c57d40cdcac82dd3520c5251795add9ae6d9ded21f7d958bd4ea663e9 Jul 07 00:08:16.117921 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.117886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" event={"ID":"3ab62ee5-f566-4887-9974-d5ebb66db81c","Type":"ContainerStarted","Data":"1f898f4c57d40cdcac82dd3520c5251795add9ae6d9ded21f7d958bd4ea663e9"} Jul 07 00:08:16.406996 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.406912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:16.489382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.489337 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd"] Jul 07 00:08:16.493478 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.493454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.495947 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.495768 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-76a59d65-serving-cert\"" Jul 07 00:08:16.495947 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.495819 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:08:16.502000 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.501965 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd"] Jul 07 00:08:16.507079 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.506947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:16.579795 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.579738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:16.658793 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.658713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49h5\" (UniqueName: \"kubernetes.io/projected/c956d493-3867-4d4a-b9a4-cdd076cd485c-kube-api-access-j49h5\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.658793 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.658755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c956d493-3867-4d4a-b9a4-cdd076cd485c-openshift-service-ca\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.658997 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.658828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c956d493-3867-4d4a-b9a4-cdd076cd485c-maas-api-tls\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.681339 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.681307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:16.759783 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.759751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49h5\" (UniqueName: \"kubernetes.io/projected/c956d493-3867-4d4a-b9a4-cdd076cd485c-kube-api-access-j49h5\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.759783 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.759790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c956d493-3867-4d4a-b9a4-cdd076cd485c-openshift-service-ca\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.760035 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.759818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c956d493-3867-4d4a-b9a4-cdd076cd485c-maas-api-tls\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.760359 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.760339 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c956d493-3867-4d4a-b9a4-cdd076cd485c-openshift-service-ca\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.762190 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.762165 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c956d493-3867-4d4a-b9a4-cdd076cd485c-maas-api-tls\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.773395 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.773372 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49h5\" (UniqueName: \"kubernetes.io/projected/c956d493-3867-4d4a-b9a4-cdd076cd485c-kube-api-access-j49h5\") pod \"maas-api-e2e-ait-76a59d65-859897f9d-xdbmd\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.811289 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.811257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:16.982147 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:16.982114 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd"] Jul 07 00:08:16.984438 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:08:16.984412 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc956d493_3867_4d4a_b9a4_cdd076cd485c.slice/crio-a9e5c19a327d2997d5dc78116639dcedc6b37931e9d1c5966174ae8f02e3f660 WatchSource:0}: Error finding container a9e5c19a327d2997d5dc78116639dcedc6b37931e9d1c5966174ae8f02e3f660: Status 404 returned error can't find the container with id a9e5c19a327d2997d5dc78116639dcedc6b37931e9d1c5966174ae8f02e3f660 Jul 07 00:08:17.126259 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:17.126216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" event={"ID":"c956d493-3867-4d4a-b9a4-cdd076cd485c","Type":"ContainerStarted","Data":"a9e5c19a327d2997d5dc78116639dcedc6b37931e9d1c5966174ae8f02e3f660"} Jul 07 00:08:17.184914 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:17.184877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:17.276315 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:17.276280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:18.133170 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:18.133122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" event={"ID":"c956d493-3867-4d4a-b9a4-cdd076cd485c","Type":"ContainerStarted","Data":"47da1f3822cd8dbf0e43e5fbb4caf6b3f5a28eec17f1544401733433b18e463b"} Jul 07 00:08:18.133599 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:18.133249 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:18.151753 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:18.151704 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" podStartSLOduration=1.8372965350000001 podStartE2EDuration="2.151687333s" totalImagesPullingTime="314.390798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:08:18.149219094 +0000 UTC m=+1528.176124335" watchObservedRunningTime="2026-07-07 00:08:18.151687333 +0000 UTC m=+1528.178592563" Jul 07 00:08:18.380064 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:18.380018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:18.878642 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:18.878589 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:08:18.878733 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:18.878682 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:08:18.878733 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:18.878712 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:08:19.140004 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:19.139916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" event={"ID":"3ab62ee5-f566-4887-9974-d5ebb66db81c","Type":"ContainerStarted","Data":"99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6"} Jul 07 00:08:19.160896 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:19.160851 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" podStartSLOduration=1.017382609 podStartE2EDuration="4.160835485s" totalImagesPullingTime="3.143452876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:08:19.158624719 +0000 UTC m=+1529.185529948" watchObservedRunningTime="2026-07-07 00:08:19.160835485 +0000 UTC m=+1529.187740760" Jul 07 00:08:19.591145 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:19.591109 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:19.595934 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:19.595908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:20.145038 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:20.145009 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:20.145896 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:20.145875 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:21.607806 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:21.607772 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:22.011043 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.010966 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd"] Jul 07 00:08:22.011214 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.011188 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" podUID="c956d493-3867-4d4a-b9a4-cdd076cd485c" containerName="maas-api" containerID="cri-o://47da1f3822cd8dbf0e43e5fbb4caf6b3f5a28eec17f1544401733433b18e463b" gracePeriod=30 Jul 07 00:08:22.041361 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.041192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:22.041539 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.041480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:22.156040 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.156006 2564 generic.go:356] "Generic (PLEG): container finished" podID="c956d493-3867-4d4a-b9a4-cdd076cd485c" containerID="47da1f3822cd8dbf0e43e5fbb4caf6b3f5a28eec17f1544401733433b18e463b" exitCode=0 Jul 07 00:08:22.156192 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.156076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" event={"ID":"c956d493-3867-4d4a-b9a4-cdd076cd485c","Type":"ContainerDied","Data":"47da1f3822cd8dbf0e43e5fbb4caf6b3f5a28eec17f1544401733433b18e463b"} Jul 07 00:08:22.298923 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.298901 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:22.301760 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.301740 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c956d493-3867-4d4a-b9a4-cdd076cd485c-maas-api-tls\") pod \"c956d493-3867-4d4a-b9a4-cdd076cd485c\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " Jul 07 00:08:22.301882 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.301828 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c956d493-3867-4d4a-b9a4-cdd076cd485c-openshift-service-ca\") pod \"c956d493-3867-4d4a-b9a4-cdd076cd485c\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " Jul 07 00:08:22.301949 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.301879 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j49h5\" (UniqueName: \"kubernetes.io/projected/c956d493-3867-4d4a-b9a4-cdd076cd485c-kube-api-access-j49h5\") pod \"c956d493-3867-4d4a-b9a4-cdd076cd485c\" (UID: \"c956d493-3867-4d4a-b9a4-cdd076cd485c\") " Jul 07 00:08:22.302210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.302184 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c956d493-3867-4d4a-b9a4-cdd076cd485c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c956d493-3867-4d4a-b9a4-cdd076cd485c" (UID: "c956d493-3867-4d4a-b9a4-cdd076cd485c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:08:22.303769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.303737 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c956d493-3867-4d4a-b9a4-cdd076cd485c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c956d493-3867-4d4a-b9a4-cdd076cd485c" (UID: "c956d493-3867-4d4a-b9a4-cdd076cd485c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:08:22.303894 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.303812 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c956d493-3867-4d4a-b9a4-cdd076cd485c-kube-api-access-j49h5" (OuterVolumeSpecName: "kube-api-access-j49h5") pod "c956d493-3867-4d4a-b9a4-cdd076cd485c" (UID: "c956d493-3867-4d4a-b9a4-cdd076cd485c"). InnerVolumeSpecName "kube-api-access-j49h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:22.402986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.402955 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c956d493-3867-4d4a-b9a4-cdd076cd485c-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:22.402986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.402979 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c956d493-3867-4d4a-b9a4-cdd076cd485c-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:22.402986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:22.402989 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49h5\" (UniqueName: \"kubernetes.io/projected/c956d493-3867-4d4a-b9a4-cdd076cd485c-kube-api-access-j49h5\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:23.163065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:23.163032 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" Jul 07 00:08:23.163550 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:23.163061 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd" event={"ID":"c956d493-3867-4d4a-b9a4-cdd076cd485c","Type":"ContainerDied","Data":"a9e5c19a327d2997d5dc78116639dcedc6b37931e9d1c5966174ae8f02e3f660"} Jul 07 00:08:23.163550 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:23.163137 2564 scope.go:117] "RemoveContainer" containerID="47da1f3822cd8dbf0e43e5fbb4caf6b3f5a28eec17f1544401733433b18e463b" Jul 07 00:08:23.179055 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:23.179021 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd"] Jul 07 00:08:23.182749 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:23.182728 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-76a59d65-859897f9d-xdbmd"] Jul 07 00:08:24.655497 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:24.655404 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c956d493-3867-4d4a-b9a4-cdd076cd485c" path="/var/lib/kubelet/pods/c956d493-3867-4d4a-b9a4-cdd076cd485c/volumes" Jul 07 00:08:27.780906 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.780872 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44"] Jul 07 00:08:27.781417 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.781223 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" podUID="3ab62ee5-f566-4887-9974-d5ebb66db81c" containerName="istio-proxy" containerID="cri-o://99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6" gracePeriod=30 Jul 07 00:08:27.803688 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.803657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:27.905919 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.905878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:27.926467 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.926443 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:27.951582 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951549 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gcfq\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-kube-api-access-4gcfq\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951750 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951643 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-certs\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951750 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951672 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-token\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951750 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951705 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-socket\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951933 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951753 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-data\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951933 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-envoy\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951933 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951807 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab62ee5-f566-4887-9974-d5ebb66db81c-istiod-ca-cert\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951933 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951863 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-podinfo\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.951933 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.951903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-credential-socket\") pod \"3ab62ee5-f566-4887-9974-d5ebb66db81c\" (UID: \"3ab62ee5-f566-4887-9974-d5ebb66db81c\") " Jul 07 00:08:27.952378 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.952350 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:27.953354 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.953293 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:27.953354 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.953324 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:27.953769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.953717 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:27.954098 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.954058 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab62ee5-f566-4887-9974-d5ebb66db81c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:08:27.955188 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.955162 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:27.955302 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.955237 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:08:27.956083 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.956057 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-kube-api-access-4gcfq" (OuterVolumeSpecName: "kube-api-access-4gcfq") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "kube-api-access-4gcfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:27.956729 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:27.956703 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ab62ee5-f566-4887-9974-d5ebb66db81c" (UID: "3ab62ee5-f566-4887-9974-d5ebb66db81c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:28.053023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.052958 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.052982 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.052991 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.052999 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.053007 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.053016 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab62ee5-f566-4887-9974-d5ebb66db81c-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.053023 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab62ee5-f566-4887-9974-d5ebb66db81c-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053320 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.053031 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab62ee5-f566-4887-9974-d5ebb66db81c-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.053320 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.053041 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcfq\" (UniqueName: \"kubernetes.io/projected/3ab62ee5-f566-4887-9974-d5ebb66db81c-kube-api-access-4gcfq\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:28.195236 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.195190 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ab62ee5-f566-4887-9974-d5ebb66db81c" containerID="99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6" exitCode=0 Jul 07 00:08:28.195422 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.195241 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" Jul 07 00:08:28.195422 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.195247 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" event={"ID":"3ab62ee5-f566-4887-9974-d5ebb66db81c","Type":"ContainerDied","Data":"99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6"} Jul 07 00:08:28.195422 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.195290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44" event={"ID":"3ab62ee5-f566-4887-9974-d5ebb66db81c","Type":"ContainerDied","Data":"1f898f4c57d40cdcac82dd3520c5251795add9ae6d9ded21f7d958bd4ea663e9"} Jul 07 00:08:28.195422 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.195312 2564 scope.go:117] "RemoveContainer" containerID="99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6" Jul 07 00:08:28.205423 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.205406 2564 scope.go:117] "RemoveContainer" containerID="99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6" Jul 07 00:08:28.205730 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:08:28.205707 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6\": container with ID starting with 99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6 not found: ID does not exist" containerID="99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6" Jul 07 00:08:28.205823 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.205738 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6"} err="failed to get container status \"99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6\": rpc error: code = NotFound desc = could not find container \"99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6\": container with ID starting with 99b1dc740d82a14cce3a05898b7e02cb5f098e4698211ffb10b91d2fa46201d6 not found: ID does not exist" Jul 07 00:08:28.217901 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.217876 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44"] Jul 07 00:08:28.221399 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.221371 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-76a59d65-openshift-default-76b989965c-ndl44"] Jul 07 00:08:28.655230 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:28.655182 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ab62ee5-f566-4887-9974-d5ebb66db81c" path="/var/lib/kubelet/pods/3ab62ee5-f566-4887-9974-d5ebb66db81c/volumes" Jul 07 00:08:30.620259 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:30.620229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:33.742539 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:33.742502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:33.850687 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:33.850612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:34.005596 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.005519 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d"] Jul 07 00:08:34.006206 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.006093 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c956d493-3867-4d4a-b9a4-cdd076cd485c" containerName="maas-api" Jul 07 00:08:34.006206 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.006114 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c956d493-3867-4d4a-b9a4-cdd076cd485c" containerName="maas-api" Jul 07 00:08:34.006206 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.006136 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab62ee5-f566-4887-9974-d5ebb66db81c" containerName="istio-proxy" Jul 07 00:08:34.006206 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.006145 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab62ee5-f566-4887-9974-d5ebb66db81c" containerName="istio-proxy" Jul 07 00:08:34.006354 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.006271 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c956d493-3867-4d4a-b9a4-cdd076cd485c" containerName="maas-api" Jul 07 00:08:34.006354 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.006282 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ab62ee5-f566-4887-9974-d5ebb66db81c" containerName="istio-proxy" Jul 07 00:08:34.011393 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.011370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.014027 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.014003 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:08:34.014291 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.014274 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-aba11d4a-openshift-default-dockercfg-dmvsg\"" Jul 07 00:08:34.039601 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.039570 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d"] Jul 07 00:08:34.074017 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.073992 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:34.103445 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrcl\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-kube-api-access-bmrcl\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103445 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-token\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-socket\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-envoy\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-data\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-credential-socket\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-podinfo\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103864 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-certs\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.103864 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.103697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951804d9-8caa-4b69-8aee-2f78fee028e0-istiod-ca-cert\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.175464 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.175433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:34.204720 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.204692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-socket\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.204720 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.204725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-envoy\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.204950 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.204741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-data\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.204950 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.204767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-credential-socket\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.204950 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.204830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-podinfo\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.204950 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.204860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-certs\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205168 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951804d9-8caa-4b69-8aee-2f78fee028e0-istiod-ca-cert\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205168 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrcl\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-kube-api-access-bmrcl\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205168 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-token\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205338 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205198 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-data\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205338 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205240 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-socket\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205338 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205283 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-certs\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205573 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205546 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-credential-socket\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.205714 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.205693 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951804d9-8caa-4b69-8aee-2f78fee028e0-istiod-ca-cert\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.207972 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.207948 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-envoy\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.208082 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.208063 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-podinfo\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.216295 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.216273 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-token\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.216429 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.216404 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrcl\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-kube-api-access-bmrcl\") pod \"e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.321691 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.321655 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:34.469440 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.469367 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d"] Jul 07 00:08:34.472441 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:08:34.472419 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951804d9_8caa_4b69_8aee_2f78fee028e0.slice/crio-69801bcc61f0aa127ceb6b0ff3130f0dcc50f278b9b907a051d8c211d50c7d51 WatchSource:0}: Error finding container 69801bcc61f0aa127ceb6b0ff3130f0dcc50f278b9b907a051d8c211d50c7d51: Status 404 returned error can't find the container with id 69801bcc61f0aa127ceb6b0ff3130f0dcc50f278b9b907a051d8c211d50c7d51 Jul 07 00:08:34.475132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.475101 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:08:34.475227 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.475169 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:08:34.475227 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:34.475210 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:08:35.232586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.232554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" event={"ID":"951804d9-8caa-4b69-8aee-2f78fee028e0","Type":"ContainerStarted","Data":"ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6"} Jul 07 00:08:35.232995 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.232619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" event={"ID":"951804d9-8caa-4b69-8aee-2f78fee028e0","Type":"ContainerStarted","Data":"69801bcc61f0aa127ceb6b0ff3130f0dcc50f278b9b907a051d8c211d50c7d51"} Jul 07 00:08:35.266968 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.266888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:35.276832 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.276779 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" podStartSLOduration=2.276763527 podStartE2EDuration="2.276763527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:08:35.274832465 +0000 UTC m=+1545.301737708" watchObservedRunningTime="2026-07-07 00:08:35.276763527 +0000 UTC m=+1545.303668801" Jul 07 00:08:35.322259 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.322225 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:35.327158 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.327133 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:35.367079 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.367048 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w"] Jul 07 00:08:35.369423 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.369407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.373396 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.373375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:08:35.373623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.373609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-aba11d4a-serving-cert\"" Jul 07 00:08:35.386380 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.386359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w"] Jul 07 00:08:35.398588 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.398566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:35.415404 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.415377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a75b00c-ca2f-408a-b54b-89387d932686-openshift-service-ca\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.415531 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.415419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.415531 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.415456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl4dx\" (UniqueName: \"kubernetes.io/projected/6a75b00c-ca2f-408a-b54b-89387d932686-kube-api-access-kl4dx\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.496855 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.496768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:35.516314 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.516280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a75b00c-ca2f-408a-b54b-89387d932686-openshift-service-ca\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.516456 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.516336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.516456 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.516361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl4dx\" (UniqueName: \"kubernetes.io/projected/6a75b00c-ca2f-408a-b54b-89387d932686-kube-api-access-kl4dx\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.516542 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:08:35.516496 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-aba11d4a-serving-cert: secret "maas-api-e2e-derive-aba11d4a-serving-cert" not found Jul 07 00:08:35.516578 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:08:35.516553 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls podName:6a75b00c-ca2f-408a-b54b-89387d932686 nodeName:}" failed. No retries permitted until 2026-07-07 00:08:36.016537451 +0000 UTC m=+1546.043442657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls") pod "maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" (UID: "6a75b00c-ca2f-408a-b54b-89387d932686") : secret "maas-api-e2e-derive-aba11d4a-serving-cert" not found Jul 07 00:08:35.517033 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.517017 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a75b00c-ca2f-408a-b54b-89387d932686-openshift-service-ca\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.525967 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.525946 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl4dx\" (UniqueName: \"kubernetes.io/projected/6a75b00c-ca2f-408a-b54b-89387d932686-kube-api-access-kl4dx\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:35.674233 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:35.674198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:36.021644 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.021606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:36.024220 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.024195 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls\") pod \"maas-api-e2e-derive-aba11d4a-76884986f9-sl94w\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:36.121306 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.121228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:36.227242 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.227194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:36.243562 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.243528 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:36.244499 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.244481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:36.280796 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.280757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:36.445278 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:36.445252 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w"] Jul 07 00:08:36.447438 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:08:36.447408 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a75b00c_ca2f_408a_b54b_89387d932686.slice/crio-2553fbe63ec309aed6226ad55fd63ef27cf92c18de39c5ac2a66ef18f5a9db21 WatchSource:0}: Error finding container 2553fbe63ec309aed6226ad55fd63ef27cf92c18de39c5ac2a66ef18f5a9db21: Status 404 returned error can't find the container with id 2553fbe63ec309aed6226ad55fd63ef27cf92c18de39c5ac2a66ef18f5a9db21 Jul 07 00:08:37.249912 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:37.249813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" event={"ID":"6a75b00c-ca2f-408a-b54b-89387d932686","Type":"ContainerStarted","Data":"f3eae3d95efcd48d6e4b90d599b6cfb7f817d68039e7a5b3c93b54e62405159e"} Jul 07 00:08:37.250271 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:37.250016 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:37.250271 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:37.250056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" event={"ID":"6a75b00c-ca2f-408a-b54b-89387d932686","Type":"ContainerStarted","Data":"2553fbe63ec309aed6226ad55fd63ef27cf92c18de39c5ac2a66ef18f5a9db21"} Jul 07 00:08:37.268927 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:37.268875 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" podStartSLOduration=1.9397875020000002 podStartE2EDuration="2.268857301s" totalImagesPullingTime="329.069799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:08:37.268309849 +0000 UTC m=+1547.295215078" watchObservedRunningTime="2026-07-07 00:08:37.268857301 +0000 UTC m=+1547.295762532" Jul 07 00:08:37.460299 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:37.460269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:39.547061 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:39.547024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:39.714474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:39.714438 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d"] Jul 07 00:08:39.714736 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:39.714700 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" podUID="951804d9-8caa-4b69-8aee-2f78fee028e0" containerName="istio-proxy" containerID="cri-o://ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6" gracePeriod=30 Jul 07 00:08:39.742987 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:39.742931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:39.847947 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:39.847863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:39.993489 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:39.993456 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w"] Jul 07 00:08:39.993730 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:39.993704 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" podUID="6a75b00c-ca2f-408a-b54b-89387d932686" containerName="maas-api" containerID="cri-o://f3eae3d95efcd48d6e4b90d599b6cfb7f817d68039e7a5b3c93b54e62405159e" gracePeriod=30 Jul 07 00:08:40.042652 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.030007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:40.265891 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.265858 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a75b00c-ca2f-408a-b54b-89387d932686" containerID="f3eae3d95efcd48d6e4b90d599b6cfb7f817d68039e7a5b3c93b54e62405159e" exitCode=0 Jul 07 00:08:40.266048 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.265936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" event={"ID":"6a75b00c-ca2f-408a-b54b-89387d932686","Type":"ContainerDied","Data":"f3eae3d95efcd48d6e4b90d599b6cfb7f817d68039e7a5b3c93b54e62405159e"} Jul 07 00:08:40.293856 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.293836 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:40.363560 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.363465 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls\") pod \"6a75b00c-ca2f-408a-b54b-89387d932686\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " Jul 07 00:08:40.363761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.363591 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a75b00c-ca2f-408a-b54b-89387d932686-openshift-service-ca\") pod \"6a75b00c-ca2f-408a-b54b-89387d932686\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " Jul 07 00:08:40.363761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.363670 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl4dx\" (UniqueName: \"kubernetes.io/projected/6a75b00c-ca2f-408a-b54b-89387d932686-kube-api-access-kl4dx\") pod \"6a75b00c-ca2f-408a-b54b-89387d932686\" (UID: \"6a75b00c-ca2f-408a-b54b-89387d932686\") " Jul 07 00:08:40.363977 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.363950 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a75b00c-ca2f-408a-b54b-89387d932686-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a75b00c-ca2f-408a-b54b-89387d932686" (UID: "6a75b00c-ca2f-408a-b54b-89387d932686"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:08:40.365772 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.365749 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a75b00c-ca2f-408a-b54b-89387d932686-kube-api-access-kl4dx" (OuterVolumeSpecName: "kube-api-access-kl4dx") pod "6a75b00c-ca2f-408a-b54b-89387d932686" (UID: "6a75b00c-ca2f-408a-b54b-89387d932686"). InnerVolumeSpecName "kube-api-access-kl4dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:40.365873 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.365825 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a75b00c-ca2f-408a-b54b-89387d932686" (UID: "6a75b00c-ca2f-408a-b54b-89387d932686"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:08:40.464711 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.464679 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a75b00c-ca2f-408a-b54b-89387d932686-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:40.464711 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.464706 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a75b00c-ca2f-408a-b54b-89387d932686-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:40.464711 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:40.464715 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl4dx\" (UniqueName: \"kubernetes.io/projected/6a75b00c-ca2f-408a-b54b-89387d932686-kube-api-access-kl4dx\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:41.272984 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:41.272950 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" Jul 07 00:08:41.272984 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:41.272959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w" event={"ID":"6a75b00c-ca2f-408a-b54b-89387d932686","Type":"ContainerDied","Data":"2553fbe63ec309aed6226ad55fd63ef27cf92c18de39c5ac2a66ef18f5a9db21"} Jul 07 00:08:41.273506 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:41.273071 2564 scope.go:117] "RemoveContainer" containerID="f3eae3d95efcd48d6e4b90d599b6cfb7f817d68039e7a5b3c93b54e62405159e" Jul 07 00:08:41.294510 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:41.294484 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w"] Jul 07 00:08:41.296295 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:41.296273 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-aba11d4a-76884986f9-sl94w"] Jul 07 00:08:41.308507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:41.308479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:42.655904 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:42.655870 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a75b00c-ca2f-408a-b54b-89387d932686" path="/var/lib/kubelet/pods/6a75b00c-ca2f-408a-b54b-89387d932686/volumes" Jul 07 00:08:45.864389 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:45.864356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:45.936175 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:45.936140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:46.270553 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.270496 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:46.304463 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.304424 2564 generic.go:356] "Generic (PLEG): container finished" podID="951804d9-8caa-4b69-8aee-2f78fee028e0" containerID="ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6" exitCode=0 Jul 07 00:08:46.304623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.304500 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" Jul 07 00:08:46.304623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.304510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" event={"ID":"951804d9-8caa-4b69-8aee-2f78fee028e0","Type":"ContainerDied","Data":"ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6"} Jul 07 00:08:46.304623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.304597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d" event={"ID":"951804d9-8caa-4b69-8aee-2f78fee028e0","Type":"ContainerDied","Data":"69801bcc61f0aa127ceb6b0ff3130f0dcc50f278b9b907a051d8c211d50c7d51"} Jul 07 00:08:46.304773 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.304644 2564 scope.go:117] "RemoveContainer" containerID="ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6" Jul 07 00:08:46.313381 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.313316 2564 scope.go:117] "RemoveContainer" containerID="ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6" Jul 07 00:08:46.314020 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:08:46.313959 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6\": container with ID starting with ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6 not found: ID does not exist" containerID="ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6" Jul 07 00:08:46.314234 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.314188 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6"} err="failed to get container status \"ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6\": rpc error: code = NotFound desc = could not find container \"ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6\": container with ID starting with ebecb82ed75cfb64105f32da5b6bd4ddd2d86bb47a56c9c1dcc7240ce379d4e6 not found: ID does not exist" Jul 07 00:08:46.317335 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317312 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951804d9-8caa-4b69-8aee-2f78fee028e0-istiod-ca-cert\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317417 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317357 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-envoy\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317417 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317394 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-socket\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317511 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317431 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-certs\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317511 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-token\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317511 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317499 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-podinfo\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317692 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317532 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmrcl\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-kube-api-access-bmrcl\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317692 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317599 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-credential-socket\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317692 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-data\") pod \"951804d9-8caa-4b69-8aee-2f78fee028e0\" (UID: \"951804d9-8caa-4b69-8aee-2f78fee028e0\") " Jul 07 00:08:46.317842 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317742 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951804d9-8caa-4b69-8aee-2f78fee028e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:08:46.317977 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317948 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:46.318046 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.317982 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951804d9-8caa-4b69-8aee-2f78fee028e0-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.318099 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.318067 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:46.318280 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.318226 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:46.318840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.318818 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:46.319981 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.319959 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-kube-api-access-bmrcl" (OuterVolumeSpecName: "kube-api-access-bmrcl") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "kube-api-access-bmrcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:46.320077 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.320058 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:08:46.320306 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.320271 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:08:46.320386 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.320342 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "951804d9-8caa-4b69-8aee-2f78fee028e0" (UID: "951804d9-8caa-4b69-8aee-2f78fee028e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:08:46.381652 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.381543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:46.418856 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418822 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.418856 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418853 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.418856 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418864 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.419057 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418872 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.419057 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418881 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.419057 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418889 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrcl\" (UniqueName: \"kubernetes.io/projected/951804d9-8caa-4b69-8aee-2f78fee028e0-kube-api-access-bmrcl\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.419057 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418899 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.419057 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.418907 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951804d9-8caa-4b69-8aee-2f78fee028e0-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:08:46.633336 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.633247 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d"] Jul 07 00:08:46.635536 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.635510 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-aba11d4a-openshift-default-7c764799f-twr4d"] Jul 07 00:08:46.654794 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.654764 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951804d9-8caa-4b69-8aee-2f78fee028e0" path="/var/lib/kubelet/pods/951804d9-8caa-4b69-8aee-2f78fee028e0/volumes" Jul 07 00:08:46.777561 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.777526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:46.877720 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:46.877686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:48.001076 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:48.001040 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:54.030860 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:54.030804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:54.132399 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:54.132347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:54.232471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:54.232432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:54.326228 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:54.326138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:54.798147 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:08:54.798112 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:20.762078 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:20.762042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:26.593936 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:26.593901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:26.672739 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:26.672703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:26.764769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:26.764735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:26.862007 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:26.861923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:42.713362 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:42.713238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:46.041007 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:46.040968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:48.596355 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:48.596309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:48.668166 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:48.668134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:48.755482 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:48.755454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:48.856308 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:48.856216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:59.375509 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:09:59.375469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:00.204337 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:00.204302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:05.989246 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:05.989214 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:06.058112 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:06.058033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:06.145796 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:06.145758 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:06.264842 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:06.264754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:06.420949 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:06.420919 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:07.060370 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:07.060341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:07.163690 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:07.163660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:07.288601 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:07.288567 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:08.862107 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:08.862060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:14.912257 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:14.912221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:15.022515 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:15.022474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:15.681246 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:15.681211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:21.481034 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:21.481000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:21.547470 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:21.547437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:21.739620 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:21.739533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:22.020887 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.020798 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7"] Jul 07 00:10:22.021575 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.021540 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951804d9-8caa-4b69-8aee-2f78fee028e0" containerName="istio-proxy" Jul 07 00:10:22.021575 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.021565 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="951804d9-8caa-4b69-8aee-2f78fee028e0" containerName="istio-proxy" Jul 07 00:10:22.021769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.021595 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a75b00c-ca2f-408a-b54b-89387d932686" containerName="maas-api" Jul 07 00:10:22.021769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.021604 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a75b00c-ca2f-408a-b54b-89387d932686" containerName="maas-api" Jul 07 00:10:22.021769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.021742 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a75b00c-ca2f-408a-b54b-89387d932686" containerName="maas-api" Jul 07 00:10:22.021769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.021755 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="951804d9-8caa-4b69-8aee-2f78fee028e0" containerName="istio-proxy" Jul 07 00:10:22.024044 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.024021 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.026348 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.026328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:10:22.026472 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.026449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-39687d3b-openshift-default-dockercfg-tv426\"" Jul 07 00:10:22.048494 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.048459 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7"] Jul 07 00:10:22.049106 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.049082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:22.136808 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.136772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-data\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137014 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.136817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d2df1e-45a9-4490-a278-1855944a2db0-istio-podinfo\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137014 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.136845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d2df1e-45a9-4490-a278-1855944a2db0-istiod-ca-cert\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137014 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.136955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-envoy\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137191 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.137016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-credential-socket\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137191 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.137050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-certs\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137191 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.137088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-socket\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137191 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.137123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbtd\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-kube-api-access-jsbtd\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.137191 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.137166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-istio-token\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238088 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-certs\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238088 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-socket\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238444 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbtd\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-kube-api-access-jsbtd\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238444 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-istio-token\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238444 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-data\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238610 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d2df1e-45a9-4490-a278-1855944a2db0-istio-podinfo\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238610 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d2df1e-45a9-4490-a278-1855944a2db0-istiod-ca-cert\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238610 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238564 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-socket\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238610 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238595 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-certs\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238610 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-envoy\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238827 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238675 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-data\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.238827 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-credential-socket\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.239019 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.238990 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-credential-socket\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.239266 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.239247 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d2df1e-45a9-4490-a278-1855944a2db0-istiod-ca-cert\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.240885 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.240858 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-envoy\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.241118 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.241101 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d2df1e-45a9-4490-a278-1855944a2db0-istio-podinfo\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.246247 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.246227 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbtd\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-kube-api-access-jsbtd\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.246494 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.246477 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-istio-token\") pod \"e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.335009 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.334970 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:22.379798 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.379761 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:22.475800 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.475775 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7"] Jul 07 00:10:22.478453 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:10:22.478421 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d2df1e_45a9_4490_a278_1855944a2db0.slice/crio-babb5885fe64da3ac8d1685569528a4f99b4b4d65bfe83643998b605d28512bc WatchSource:0}: Error finding container babb5885fe64da3ac8d1685569528a4f99b4b4d65bfe83643998b605d28512bc: Status 404 returned error can't find the container with id babb5885fe64da3ac8d1685569528a4f99b4b4d65bfe83643998b605d28512bc Jul 07 00:10:22.480800 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.480758 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:10:22.480914 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.480852 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:10:22.480955 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.480920 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:10:22.761050 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.760961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" event={"ID":"78d2df1e-45a9-4490-a278-1855944a2db0","Type":"ContainerStarted","Data":"c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d"} Jul 07 00:10:22.761050 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.761024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" event={"ID":"78d2df1e-45a9-4490-a278-1855944a2db0","Type":"ContainerStarted","Data":"babb5885fe64da3ac8d1685569528a4f99b4b4d65bfe83643998b605d28512bc"} Jul 07 00:10:22.780128 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:22.780077 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" podStartSLOduration=0.780059593 podStartE2EDuration="780.059593ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:10:22.778524061 +0000 UTC m=+1652.805429292" watchObservedRunningTime="2026-07-07 00:10:22.780059593 +0000 UTC m=+1652.806964822" Jul 07 00:10:23.335509 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:23.335470 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:23.341376 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:23.341351 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:23.766779 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:23.766697 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:23.767566 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:23.767549 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:24.846994 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:24.846952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:24.935315 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:24.935283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:24.991821 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:24.991773 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln"] Jul 07 00:10:25.001053 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.001024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.006275 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.006243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-39687d3b-serving-cert\"" Jul 07 00:10:25.006434 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.006415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:10:25.008210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.008184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln"] Jul 07 00:10:25.026575 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.026542 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:25.165759 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.165659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-openshift-service-ca\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.165759 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.165701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.165759 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.165725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgfs2\" (UniqueName: \"kubernetes.io/projected/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-kube-api-access-jgfs2\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.266503 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.266467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-openshift-service-ca\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.266503 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.266506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.266749 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.266532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfs2\" (UniqueName: \"kubernetes.io/projected/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-kube-api-access-jgfs2\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.266749 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:10:25.266665 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-39687d3b-serving-cert: secret "maas-api-e2e-mt-39687d3b-serving-cert" not found Jul 07 00:10:25.266749 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:10:25.266745 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls podName:1227de3d-2e3c-4b39-9a27-a57b62c7ee43 nodeName:}" failed. No retries permitted until 2026-07-07 00:10:25.766722772 +0000 UTC m=+1655.793627984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls") pod "maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" (UID: "1227de3d-2e3c-4b39-9a27-a57b62c7ee43") : secret "maas-api-e2e-mt-39687d3b-serving-cert" not found Jul 07 00:10:25.267231 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.267203 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-openshift-service-ca\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.276131 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.276101 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfs2\" (UniqueName: \"kubernetes.io/projected/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-kube-api-access-jgfs2\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.503824 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.503740 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:25.618397 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.618362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:25.733821 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.733793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:25.771607 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.771517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.774229 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.774202 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls\") pod \"maas-api-e2e-mt-39687d3b-549899bdd8-2gnln\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:25.917171 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:25.917139 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:26.068896 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:26.068867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln"] Jul 07 00:10:26.070817 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:10:26.070783 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1227de3d_2e3c_4b39_9a27_a57b62c7ee43.slice/crio-9bfa8d2f761d5ccb6c72ee9628ca53c73afc8fccf024f8a63c8066ee112c801d WatchSource:0}: Error finding container 9bfa8d2f761d5ccb6c72ee9628ca53c73afc8fccf024f8a63c8066ee112c801d: Status 404 returned error can't find the container with id 9bfa8d2f761d5ccb6c72ee9628ca53c73afc8fccf024f8a63c8066ee112c801d Jul 07 00:10:26.783957 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:26.783919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" event={"ID":"1227de3d-2e3c-4b39-9a27-a57b62c7ee43","Type":"ContainerStarted","Data":"c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0"} Jul 07 00:10:26.784145 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:26.783973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" event={"ID":"1227de3d-2e3c-4b39-9a27-a57b62c7ee43","Type":"ContainerStarted","Data":"9bfa8d2f761d5ccb6c72ee9628ca53c73afc8fccf024f8a63c8066ee112c801d"} Jul 07 00:10:26.784145 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:26.784017 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:26.801622 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:26.801570 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" podStartSLOduration=2.360508581 podStartE2EDuration="2.801549532s" totalImagesPullingTime="441.040951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:10:26.800194541 +0000 UTC m=+1656.827099771" watchObservedRunningTime="2026-07-07 00:10:26.801549532 +0000 UTC m=+1656.828454792" Jul 07 00:10:29.393463 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:29.393430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:30.125330 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:30.125297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:30.215748 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:30.215668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:30.554514 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:30.554476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:32.793945 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:32.793913 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:36.523395 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:36.523361 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:36.637717 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:36.637682 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:36.991296 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:36.991266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:37.435580 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.435496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:37.477822 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.477789 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln"] Jul 07 00:10:37.478005 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.477982 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" podUID="1227de3d-2e3c-4b39-9a27-a57b62c7ee43" containerName="maas-api" containerID="cri-o://c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0" gracePeriod=30 Jul 07 00:10:37.504775 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.504747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:37.726287 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.726266 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:37.795909 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.795881 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgfs2\" (UniqueName: \"kubernetes.io/projected/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-kube-api-access-jgfs2\") pod \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " Jul 07 00:10:37.796100 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.796013 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls\") pod \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " Jul 07 00:10:37.796100 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.796049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-openshift-service-ca\") pod \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\" (UID: \"1227de3d-2e3c-4b39-9a27-a57b62c7ee43\") " Jul 07 00:10:37.796496 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.796457 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1227de3d-2e3c-4b39-9a27-a57b62c7ee43" (UID: "1227de3d-2e3c-4b39-9a27-a57b62c7ee43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:10:37.797929 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.797901 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1227de3d-2e3c-4b39-9a27-a57b62c7ee43" (UID: "1227de3d-2e3c-4b39-9a27-a57b62c7ee43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:10:37.798010 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.797954 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-kube-api-access-jgfs2" (OuterVolumeSpecName: "kube-api-access-jgfs2") pod "1227de3d-2e3c-4b39-9a27-a57b62c7ee43" (UID: "1227de3d-2e3c-4b39-9a27-a57b62c7ee43"). InnerVolumeSpecName "kube-api-access-jgfs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:10:37.841358 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.841324 2564 generic.go:356] "Generic (PLEG): container finished" podID="78d2df1e-45a9-4490-a278-1855944a2db0" containerID="c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d" exitCode=0 Jul 07 00:10:37.841523 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.841402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" event={"ID":"78d2df1e-45a9-4490-a278-1855944a2db0","Type":"ContainerDied","Data":"c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d"} Jul 07 00:10:37.841870 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.841850 2564 scope.go:117] "RemoveContainer" containerID="c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d" Jul 07 00:10:37.842772 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.842746 2564 generic.go:356] "Generic (PLEG): container finished" podID="1227de3d-2e3c-4b39-9a27-a57b62c7ee43" containerID="c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0" exitCode=0 Jul 07 00:10:37.842870 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.842773 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" Jul 07 00:10:37.842870 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.842818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" event={"ID":"1227de3d-2e3c-4b39-9a27-a57b62c7ee43","Type":"ContainerDied","Data":"c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0"} Jul 07 00:10:37.842870 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.842866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln" event={"ID":"1227de3d-2e3c-4b39-9a27-a57b62c7ee43","Type":"ContainerDied","Data":"9bfa8d2f761d5ccb6c72ee9628ca53c73afc8fccf024f8a63c8066ee112c801d"} Jul 07 00:10:37.842989 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.842896 2564 scope.go:117] "RemoveContainer" containerID="c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0" Jul 07 00:10:37.842989 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.842929 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:10:37.843063 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.843012 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:10:37.843116 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.843062 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:10:37.853891 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.853875 2564 scope.go:117] "RemoveContainer" containerID="c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0" Jul 07 00:10:37.854146 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:10:37.854123 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0\": container with ID starting with c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0 not found: ID does not exist" containerID="c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0" Jul 07 00:10:37.854198 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.854160 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0"} err="failed to get container status \"c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0\": rpc error: code = NotFound desc = could not find container \"c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0\": container with ID starting with c125c33ac13b6c199180c230e6641a4cfe63db11eeef30cc6e1caa57f1c6bde0 not found: ID does not exist" Jul 07 00:10:37.882796 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.882771 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln"] Jul 07 00:10:37.889773 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.889747 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-39687d3b-549899bdd8-2gnln"] Jul 07 00:10:37.897858 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.897831 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:37.897986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.897863 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:37.897986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:37.897878 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgfs2\" (UniqueName: \"kubernetes.io/projected/1227de3d-2e3c-4b39-9a27-a57b62c7ee43-kube-api-access-jgfs2\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:38.654842 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:38.654808 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1227de3d-2e3c-4b39-9a27-a57b62c7ee43" path="/var/lib/kubelet/pods/1227de3d-2e3c-4b39-9a27-a57b62c7ee43/volumes" Jul 07 00:10:38.767680 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:38.767623 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:38.850609 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:38.850576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" event={"ID":"78d2df1e-45a9-4490-a278-1855944a2db0","Type":"ContainerStarted","Data":"414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4"} Jul 07 00:10:39.336411 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:39.336372 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:39.337745 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:39.337713 2564 patch_prober.go:28] interesting pod/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 00:10:39.337988 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:39.337957 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 00:10:40.335875 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:40.335829 2564 patch_prober.go:28] interesting pod/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 00:10:40.336383 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:40.335899 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 00:10:41.336005 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:41.335962 2564 patch_prober.go:28] interesting pod/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 00:10:41.336486 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:41.336033 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 00:10:42.335601 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:42.335556 2564 patch_prober.go:28] interesting pod/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 00:10:42.335828 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:42.335660 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 00:10:43.322042 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.322006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:43.340066 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.340032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:43.340412 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.340391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:43.341038 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.341019 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:43.383241 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.383199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:43.572395 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.572310 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7"] Jul 07 00:10:43.603469 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.603432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:43.820875 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.820843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:43.874566 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.874472 2564 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" secret="" err="secret \"e2e-mt-39687d3b-openshift-default-dockercfg-tv426\" not found" Jul 07 00:10:43.958771 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:43.958737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:44.051808 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:44.051778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:44.194187 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:44.194106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:44.879556 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:44.879512 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" containerID="cri-o://414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4" gracePeriod=30 Jul 07 00:10:50.138440 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.138418 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:50.220096 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220011 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-data\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220240 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d2df1e-45a9-4490-a278-1855944a2db0-istiod-ca-cert\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220240 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220162 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-socket\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220240 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220199 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d2df1e-45a9-4490-a278-1855944a2db0-istio-podinfo\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220360 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220241 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-envoy\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220398 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220355 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsbtd\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-kube-api-access-jsbtd\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220433 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220404 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-certs\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220433 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220423 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-credential-socket\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.220493 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.220441 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-istio-token\") pod \"78d2df1e-45a9-4490-a278-1855944a2db0\" (UID: \"78d2df1e-45a9-4490-a278-1855944a2db0\") " Jul 07 00:10:50.224067 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.223896 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:10:50.224067 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.223932 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d2df1e-45a9-4490-a278-1855944a2db0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:10:50.224067 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.224011 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:10:50.224067 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.224020 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:10:50.224484 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.224459 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-data" (OuterVolumeSpecName: "istio-data") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:10:50.225905 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.225868 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:10:50.226008 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.225936 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-istio-token" (OuterVolumeSpecName: "istio-token") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:10:50.226008 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.225940 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-kube-api-access-jsbtd" (OuterVolumeSpecName: "kube-api-access-jsbtd") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "kube-api-access-jsbtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:10:50.226008 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.225991 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78d2df1e-45a9-4490-a278-1855944a2db0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78d2df1e-45a9-4490-a278-1855944a2db0" (UID: "78d2df1e-45a9-4490-a278-1855944a2db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:10:50.234215 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.234194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:50.303245 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.303196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:50.322009 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.321980 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322009 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322005 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d2df1e-45a9-4490-a278-1855944a2db0-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322009 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322015 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322024 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d2df1e-45a9-4490-a278-1855944a2db0-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322035 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322044 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbtd\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-kube-api-access-jsbtd\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322053 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322063 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d2df1e-45a9-4490-a278-1855944a2db0-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.322210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.322070 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d2df1e-45a9-4490-a278-1855944a2db0-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:10:50.389769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.389737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:50.492887 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.492806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:50.593878 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.593844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:50.910494 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.910459 2564 generic.go:356] "Generic (PLEG): container finished" podID="78d2df1e-45a9-4490-a278-1855944a2db0" containerID="414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4" exitCode=0 Jul 07 00:10:50.910695 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.910516 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" Jul 07 00:10:50.910695 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.910541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" event={"ID":"78d2df1e-45a9-4490-a278-1855944a2db0","Type":"ContainerDied","Data":"414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4"} Jul 07 00:10:50.910695 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.910595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7" event={"ID":"78d2df1e-45a9-4490-a278-1855944a2db0","Type":"ContainerDied","Data":"babb5885fe64da3ac8d1685569528a4f99b4b4d65bfe83643998b605d28512bc"} Jul 07 00:10:50.910695 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.910617 2564 scope.go:117] "RemoveContainer" containerID="414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4" Jul 07 00:10:50.922115 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.922093 2564 scope.go:117] "RemoveContainer" containerID="c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d" Jul 07 00:10:50.930999 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.930979 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7"] Jul 07 00:10:50.931749 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.931734 2564 scope.go:117] "RemoveContainer" containerID="414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4" Jul 07 00:10:50.932010 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:10:50.931982 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4\": container with ID starting with 414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4 not found: ID does not exist" containerID="414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4" Jul 07 00:10:50.932132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.932021 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4"} err="failed to get container status \"414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4\": rpc error: code = NotFound desc = could not find container \"414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4\": container with ID starting with 414baf341d44bd71d57f37a1c6187b9ff357f7ade87561142f5d8908851168c4 not found: ID does not exist" Jul 07 00:10:50.932132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.932042 2564 scope.go:117] "RemoveContainer" containerID="c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d" Jul 07 00:10:50.932390 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:10:50.932365 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d\": container with ID starting with c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d not found: ID does not exist" containerID="c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d" Jul 07 00:10:50.932523 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.932405 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d"} err="failed to get container status \"c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d\": rpc error: code = NotFound desc = could not find container \"c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d\": container with ID starting with c094881cab343fdd46b23491af49bd8292ff9792c92033b0bc7038e375e3b64d not found: ID does not exist" Jul 07 00:10:50.935057 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:50.935039 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-39687d3b-openshift-default-5869f9557f-g24x7"] Jul 07 00:10:52.654590 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:52.654553 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" path="/var/lib/kubelet/pods/78d2df1e-45a9-4490-a278-1855944a2db0/volumes" Jul 07 00:10:56.277959 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:56.277918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:10:56.339977 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:10:56.339928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:03.208365 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.208303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:03.529874 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.529840 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c"] Jul 07 00:11:03.530233 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530219 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" Jul 07 00:11:03.530279 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530234 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" Jul 07 00:11:03.530279 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530245 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1227de3d-2e3c-4b39-9a27-a57b62c7ee43" containerName="maas-api" Jul 07 00:11:03.530279 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530250 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1227de3d-2e3c-4b39-9a27-a57b62c7ee43" containerName="maas-api" Jul 07 00:11:03.530279 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530264 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" Jul 07 00:11:03.530279 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530270 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" Jul 07 00:11:03.530434 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530331 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1227de3d-2e3c-4b39-9a27-a57b62c7ee43" containerName="maas-api" Jul 07 00:11:03.530434 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530341 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" Jul 07 00:11:03.530434 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.530347 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d2df1e-45a9-4490-a278-1855944a2db0" containerName="istio-proxy" Jul 07 00:11:03.532830 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.532814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.535184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.535160 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:11:03.535380 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.535366 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e68d4ac7-openshift-default-dockercfg-2w4g7\"" Jul 07 00:11:03.536298 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-socket\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536371 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-credential-socket\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536426 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-podinfo\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536481 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-certs\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536481 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-envoy\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536582 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-data\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536582 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rfx5\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-kube-api-access-9rfx5\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536582 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-token\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.536582 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.536575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d12ae2-5d3a-498a-b763-0d005d766ebf-istiod-ca-cert\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.550981 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.550946 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c"] Jul 07 00:11:03.570720 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.570687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:03.637692 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.637620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-certs\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.637883 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.637704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-envoy\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.637883 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.637733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-data\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.637883 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.637761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfx5\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-kube-api-access-9rfx5\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.637883 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.637800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-token\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.637883 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.637846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d12ae2-5d3a-498a-b763-0d005d766ebf-istiod-ca-cert\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.637883 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.637878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-socket\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.638197 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.638062 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-certs\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.638197 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.638088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-credential-socket\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.638197 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.638147 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-data\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.638197 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.638175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-podinfo\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.638611 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.638588 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d12ae2-5d3a-498a-b763-0d005d766ebf-istiod-ca-cert\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.638704 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.638620 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-credential-socket\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.638763 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.638721 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-socket\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.640167 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.640144 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-envoy\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.640447 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.640432 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-podinfo\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.646295 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.646274 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-token\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.646780 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.646760 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfx5\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-kube-api-access-9rfx5\") pod \"e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.843208 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.843112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:03.995995 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:03.995972 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c"] Jul 07 00:11:03.998198 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:11:03.998173 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d12ae2_5d3a_498a_b763_0d005d766ebf.slice/crio-50e8161136bcb541e4c101c3372ca486537c882664e255e1c9599ab34c42cb5e WatchSource:0}: Error finding container 50e8161136bcb541e4c101c3372ca486537c882664e255e1c9599ab34c42cb5e: Status 404 returned error can't find the container with id 50e8161136bcb541e4c101c3372ca486537c882664e255e1c9599ab34c42cb5e Jul 07 00:11:04.000349 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:04.000313 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:11:04.000413 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:04.000384 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:11:04.000413 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:04.000410 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:11:04.016584 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:04.016560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:04.979978 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:04.979937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" event={"ID":"85d12ae2-5d3a-498a-b763-0d005d766ebf","Type":"ContainerStarted","Data":"e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4"} Jul 07 00:11:04.980383 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:04.979997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" event={"ID":"85d12ae2-5d3a-498a-b763-0d005d766ebf","Type":"ContainerStarted","Data":"50e8161136bcb541e4c101c3372ca486537c882664e255e1c9599ab34c42cb5e"} Jul 07 00:11:05.000374 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:05.000320 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" podStartSLOduration=2.000305197 podStartE2EDuration="2.000305197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:11:04.997748644 +0000 UTC m=+1695.024653872" watchObservedRunningTime="2026-07-07 00:11:05.000305197 +0000 UTC m=+1695.027210427" Jul 07 00:11:05.844341 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:05.844302 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:05.849732 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:05.849703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:05.984727 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:05.984685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:05.985753 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:05.985732 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:07.024880 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.024840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:07.140877 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.140839 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9"] Jul 07 00:11:07.143297 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.143272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.146807 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.146784 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-e68d4ac7-serving-cert\"" Jul 07 00:11:07.146947 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.146931 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:11:07.154952 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.154927 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9"] Jul 07 00:11:07.163310 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.163262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.163431 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.163313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtjz\" (UniqueName: \"kubernetes.io/projected/1af22d90-4193-45c7-ab56-b7120ae530a2-kube-api-access-nrtjz\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.163431 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.163382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af22d90-4193-45c7-ab56-b7120ae530a2-openshift-service-ca\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.181315 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.181287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:07.264663 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.264612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.264868 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.264684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtjz\" (UniqueName: \"kubernetes.io/projected/1af22d90-4193-45c7-ab56-b7120ae530a2-kube-api-access-nrtjz\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.264868 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.264756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af22d90-4193-45c7-ab56-b7120ae530a2-openshift-service-ca\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.264868 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:11:07.264773 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-e68d4ac7-serving-cert: secret "maas-api-e2e-mt-e68d4ac7-serving-cert" not found Jul 07 00:11:07.264868 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:11:07.264857 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls podName:1af22d90-4193-45c7-ab56-b7120ae530a2 nodeName:}" failed. No retries permitted until 2026-07-07 00:11:07.764835566 +0000 UTC m=+1697.791740776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls") pod "maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" (UID: "1af22d90-4193-45c7-ab56-b7120ae530a2") : secret "maas-api-e2e-mt-e68d4ac7-serving-cert" not found Jul 07 00:11:07.265426 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.265403 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af22d90-4193-45c7-ab56-b7120ae530a2-openshift-service-ca\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.275459 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.275431 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtjz\" (UniqueName: \"kubernetes.io/projected/1af22d90-4193-45c7-ab56-b7120ae530a2-kube-api-access-nrtjz\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.357016 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.356974 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:07.702118 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.701995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:07.769084 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.769052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.771344 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.771317 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls\") pod \"maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:07.805705 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:07.805654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:08.055182 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:08.055142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:08.412319 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:08.412292 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9"] Jul 07 00:11:08.414744 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:11:08.414712 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af22d90_4193_45c7_ab56_b7120ae530a2.slice/crio-d69b7564abcf8a72fa9d3d0fa3b7c565e764cc036b416d7bebae0fc530fd357c WatchSource:0}: Error finding container d69b7564abcf8a72fa9d3d0fa3b7c565e764cc036b416d7bebae0fc530fd357c: Status 404 returned error can't find the container with id d69b7564abcf8a72fa9d3d0fa3b7c565e764cc036b416d7bebae0fc530fd357c Jul 07 00:11:09.001675 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:09.001576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" event={"ID":"1af22d90-4193-45c7-ab56-b7120ae530a2","Type":"ContainerStarted","Data":"27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295"} Jul 07 00:11:09.001675 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:09.001641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" event={"ID":"1af22d90-4193-45c7-ab56-b7120ae530a2","Type":"ContainerStarted","Data":"d69b7564abcf8a72fa9d3d0fa3b7c565e764cc036b416d7bebae0fc530fd357c"} Jul 07 00:11:09.001675 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:09.001669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:09.019498 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:09.019450 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" podStartSLOduration=1.730315734 podStartE2EDuration="2.019435052s" totalImagesPullingTime="289.119318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:11:09.017831721 +0000 UTC m=+1699.044736942" watchObservedRunningTime="2026-07-07 00:11:09.019435052 +0000 UTC m=+1699.046340274" Jul 07 00:11:11.501754 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:11.501721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:15.011571 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:15.011492 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:11:21.300509 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:21.300468 2564 patch_prober.go:28] interesting pod/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:39630->10.132.0.54:15021: read: connection reset by peer" start-of-body= Jul 07 00:11:21.300971 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:21.300545 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:39630->10.132.0.54:15021: read: connection reset by peer" Jul 07 00:11:22.083793 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.083755 2564 generic.go:356] "Generic (PLEG): container finished" podID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerID="e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4" exitCode=0 Jul 07 00:11:22.083969 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.083824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" event={"ID":"85d12ae2-5d3a-498a-b763-0d005d766ebf","Type":"ContainerDied","Data":"e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4"} Jul 07 00:11:22.084202 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.084189 2564 scope.go:117] "RemoveContainer" containerID="e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4" Jul 07 00:11:22.085254 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.085214 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:11:22.085393 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.085300 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:11:22.085393 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.085352 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:11:22.239539 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.239499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:22.336292 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:22.336208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:23.091089 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:23.091053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" event={"ID":"85d12ae2-5d3a-498a-b763-0d005d766ebf","Type":"ContainerStarted","Data":"ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559"} Jul 07 00:11:23.843644 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:23.843581 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:23.844140 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:23.843913 2564 patch_prober.go:28] interesting pod/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 00:11:23.844140 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:23.843975 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 00:11:24.844052 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:24.844014 2564 patch_prober.go:28] interesting pod/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 00:11:24.844436 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:24.844070 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 00:11:25.843698 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:25.843654 2564 patch_prober.go:28] interesting pod/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 00:11:25.843900 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:25.843722 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 00:11:26.843674 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:26.843621 2564 patch_prober.go:28] interesting pod/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 00:11:26.844056 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:26.843696 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 00:11:27.847444 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:27.847414 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:27.847912 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:27.847808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:27.848584 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:27.848561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:11:29.460590 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:29.460556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:29.561551 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:29.561514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:42.441691 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:11:42.441658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:04.544663 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:04.544608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:04.885702 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:04.885592 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9"] Jul 07 00:12:04.886207 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:04.886171 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" podUID="1af22d90-4193-45c7-ab56-b7120ae530a2" containerName="maas-api" containerID="cri-o://27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295" gracePeriod=30 Jul 07 00:12:05.007767 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.007720 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" podUID="1af22d90-4193-45c7-ab56-b7120ae530a2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.55:8443/health\": dial tcp 10.132.0.55:8443: connect: connection refused" Jul 07 00:12:05.130623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.130598 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:12:05.184091 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.184005 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls\") pod \"1af22d90-4193-45c7-ab56-b7120ae530a2\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " Jul 07 00:12:05.184091 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.184045 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af22d90-4193-45c7-ab56-b7120ae530a2-openshift-service-ca\") pod \"1af22d90-4193-45c7-ab56-b7120ae530a2\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " Jul 07 00:12:05.184091 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.184076 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrtjz\" (UniqueName: \"kubernetes.io/projected/1af22d90-4193-45c7-ab56-b7120ae530a2-kube-api-access-nrtjz\") pod \"1af22d90-4193-45c7-ab56-b7120ae530a2\" (UID: \"1af22d90-4193-45c7-ab56-b7120ae530a2\") " Jul 07 00:12:05.184471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.184447 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af22d90-4193-45c7-ab56-b7120ae530a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1af22d90-4193-45c7-ab56-b7120ae530a2" (UID: "1af22d90-4193-45c7-ab56-b7120ae530a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:12:05.186182 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.186156 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af22d90-4193-45c7-ab56-b7120ae530a2-kube-api-access-nrtjz" (OuterVolumeSpecName: "kube-api-access-nrtjz") pod "1af22d90-4193-45c7-ab56-b7120ae530a2" (UID: "1af22d90-4193-45c7-ab56-b7120ae530a2"). InnerVolumeSpecName "kube-api-access-nrtjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:12:05.186264 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.186160 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1af22d90-4193-45c7-ab56-b7120ae530a2" (UID: "1af22d90-4193-45c7-ab56-b7120ae530a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:12:05.284865 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.284828 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af22d90-4193-45c7-ab56-b7120ae530a2-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:12:05.284865 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.284859 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af22d90-4193-45c7-ab56-b7120ae530a2-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:12:05.284865 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.284870 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtjz\" (UniqueName: \"kubernetes.io/projected/1af22d90-4193-45c7-ab56-b7120ae530a2-kube-api-access-nrtjz\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:12:05.302982 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.302945 2564 generic.go:356] "Generic (PLEG): container finished" podID="1af22d90-4193-45c7-ab56-b7120ae530a2" containerID="27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295" exitCode=0 Jul 07 00:12:05.303135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.302991 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" Jul 07 00:12:05.303135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.303036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" event={"ID":"1af22d90-4193-45c7-ab56-b7120ae530a2","Type":"ContainerDied","Data":"27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295"} Jul 07 00:12:05.303135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.303089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9" event={"ID":"1af22d90-4193-45c7-ab56-b7120ae530a2","Type":"ContainerDied","Data":"d69b7564abcf8a72fa9d3d0fa3b7c565e764cc036b416d7bebae0fc530fd357c"} Jul 07 00:12:05.303135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.303111 2564 scope.go:117] "RemoveContainer" containerID="27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295" Jul 07 00:12:05.312502 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.312481 2564 scope.go:117] "RemoveContainer" containerID="27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295" Jul 07 00:12:05.312795 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:12:05.312775 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295\": container with ID starting with 27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295 not found: ID does not exist" containerID="27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295" Jul 07 00:12:05.312848 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.312805 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295"} err="failed to get container status \"27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295\": rpc error: code = NotFound desc = could not find container \"27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295\": container with ID starting with 27dbbf37815e4effbbb0bdd85ac1badab2f5b7d78772cf5743d0f61621695295 not found: ID does not exist" Jul 07 00:12:05.325496 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.325473 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9"] Jul 07 00:12:05.329574 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:05.329553 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-e68d4ac7-67bb485b7d-j48n9"] Jul 07 00:12:06.106475 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:06.106435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:06.654520 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:06.654483 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1af22d90-4193-45c7-ab56-b7120ae530a2" path="/var/lib/kubelet/pods/1af22d90-4193-45c7-ab56-b7120ae530a2/volumes" Jul 07 00:12:10.678110 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:10.678076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:15.717403 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:15.717365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:40.597908 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:40.597872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:43.752206 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:43.752120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:43.807693 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:43.807656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:50.564621 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:50.564592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:12:50.567219 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:50.567194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:12:50.568159 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:50.568143 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:12:50.570688 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:50.570667 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:12:52.353339 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:52.353307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:52.873513 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:52.873476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:58.541402 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:12:58.541357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:12.743999 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:12.743963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:12.813382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:12.813348 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:13.006581 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:13.006489 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c"] Jul 07 00:13:13.006819 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:13.006774 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" containerID="cri-o://ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559" gracePeriod=30 Jul 07 00:13:13.024658 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:13.024610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:13.268276 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:13.268198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:14.348772 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:14.348734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:14.500777 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:14.500739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:14.595908 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:14.595870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:15.338810 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:15.338771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:15.437447 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:15.437410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:15.537825 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:15.537789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:17.023945 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:17.023907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:18.263554 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.263525 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:13:18.335357 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d12ae2-5d3a-498a-b763-0d005d766ebf-istiod-ca-cert\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335556 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335377 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-podinfo\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335556 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335407 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-socket\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335556 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-data\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335768 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335600 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-credential-socket\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335768 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335687 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-envoy\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335768 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335718 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-token\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335768 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335749 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-certs\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335982 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335781 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rfx5\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-kube-api-access-9rfx5\") pod \"85d12ae2-5d3a-498a-b763-0d005d766ebf\" (UID: \"85d12ae2-5d3a-498a-b763-0d005d766ebf\") " Jul 07 00:13:18.335982 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335863 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:18.335982 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335872 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:18.335982 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.335887 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d12ae2-5d3a-498a-b763-0d005d766ebf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:13:18.336178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.336091 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d12ae2-5d3a-498a-b763-0d005d766ebf-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.336178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.336093 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:18.336178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.336109 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.336178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.336123 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.336412 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.336388 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-data" (OuterVolumeSpecName: "istio-data") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:18.337806 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.337784 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:13:18.337915 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.337888 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-token" (OuterVolumeSpecName: "istio-token") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:13:18.337980 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.337944 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:18.338023 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.337985 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-kube-api-access-9rfx5" (OuterVolumeSpecName: "kube-api-access-9rfx5") pod "85d12ae2-5d3a-498a-b763-0d005d766ebf" (UID: "85d12ae2-5d3a-498a-b763-0d005d766ebf"). InnerVolumeSpecName "kube-api-access-9rfx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:13:18.436586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.436544 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.436586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.436577 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.436586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.436588 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.436586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.436596 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.436882 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.436606 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d12ae2-5d3a-498a-b763-0d005d766ebf-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.436882 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.436614 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rfx5\" (UniqueName: \"kubernetes.io/projected/85d12ae2-5d3a-498a-b763-0d005d766ebf-kube-api-access-9rfx5\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:13:18.673159 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.673127 2564 generic.go:356] "Generic (PLEG): container finished" podID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerID="ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559" exitCode=0 Jul 07 00:13:18.673329 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.673173 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" Jul 07 00:13:18.673329 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.673184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" event={"ID":"85d12ae2-5d3a-498a-b763-0d005d766ebf","Type":"ContainerDied","Data":"ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559"} Jul 07 00:13:18.673329 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.673225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c" event={"ID":"85d12ae2-5d3a-498a-b763-0d005d766ebf","Type":"ContainerDied","Data":"50e8161136bcb541e4c101c3372ca486537c882664e255e1c9599ab34c42cb5e"} Jul 07 00:13:18.673329 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.673246 2564 scope.go:117] "RemoveContainer" containerID="ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559" Jul 07 00:13:18.682434 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.682416 2564 scope.go:117] "RemoveContainer" containerID="e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4" Jul 07 00:13:18.690713 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.690694 2564 scope.go:117] "RemoveContainer" containerID="ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559" Jul 07 00:13:18.690850 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.690829 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c"] Jul 07 00:13:18.690964 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:13:18.690943 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559\": container with ID starting with ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559 not found: ID does not exist" containerID="ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559" Jul 07 00:13:18.691014 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.690973 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559"} err="failed to get container status \"ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559\": rpc error: code = NotFound desc = could not find container \"ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559\": container with ID starting with ae1d17351560373ed7dfcb2dc899e9228dc76f67355873c90e8b5e3da5fbd559 not found: ID does not exist" Jul 07 00:13:18.691014 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.690990 2564 scope.go:117] "RemoveContainer" containerID="e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4" Jul 07 00:13:18.691195 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:13:18.691180 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4\": container with ID starting with e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4 not found: ID does not exist" containerID="e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4" Jul 07 00:13:18.691260 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.691200 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4"} err="failed to get container status \"e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4\": rpc error: code = NotFound desc = could not find container \"e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4\": container with ID starting with e6da35cf12b950ab8c9c1b096ee3a39f3ddafb0678753596e19de2edee1fb0d4 not found: ID does not exist" Jul 07 00:13:18.701874 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:18.701849 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68d4ac7-openshift-default-56675f499-rf94c"] Jul 07 00:13:20.654510 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:20.654470 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" path="/var/lib/kubelet/pods/85d12ae2-5d3a-498a-b763-0d005d766ebf/volumes" Jul 07 00:13:22.774345 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:22.774308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:22.835047 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:22.835015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:23.481676 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:23.481624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:29.420003 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:29.419963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:29.482955 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:29.482926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:29.609704 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:29.609673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:29.711867 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:29.711789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:40.272389 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:40.272351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:40.672529 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:40.672493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:13:55.557354 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:13:55.557285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:01.664886 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:01.664855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:07.486553 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:07.486513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:07.619095 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:07.619061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:07.853893 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:07.853864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:08.145231 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145143 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg"] Jul 07 00:14:08.145544 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145530 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" Jul 07 00:14:08.145544 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145544 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" Jul 07 00:14:08.145687 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145553 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af22d90-4193-45c7-ab56-b7120ae530a2" containerName="maas-api" Jul 07 00:14:08.145687 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145559 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af22d90-4193-45c7-ab56-b7120ae530a2" containerName="maas-api" Jul 07 00:14:08.145687 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145618 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" Jul 07 00:14:08.145687 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145654 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" Jul 07 00:14:08.145687 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.145663 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1af22d90-4193-45c7-ab56-b7120ae530a2" containerName="maas-api" Jul 07 00:14:08.148674 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.148652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.151154 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.151133 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0a7418-openshift-default-dockercfg-f44m8\"" Jul 07 00:14:08.151509 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.151496 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:14:08.160333 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.160307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg"] Jul 07 00:14:08.170261 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.170228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:08.273840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.273809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-socket\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274054 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.273853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-credential-socket\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274054 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.273872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c556e-6332-4919-b537-678db2c67d1d-istio-podinfo\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274054 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.273965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c556e-6332-4919-b537-678db2c67d1d-istiod-ca-cert\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274054 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.274013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-envoy\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274054 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.274038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-istio-token\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274283 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.274099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-certs\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274283 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.274122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-data\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.274283 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.274178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzd56\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-kube-api-access-hzd56\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.303493 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.303463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:08.374840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.374812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-certs\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.374840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.374842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-data\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375108 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.374886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzd56\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-kube-api-access-hzd56\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375108 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.374919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-socket\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375108 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.374950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-credential-socket\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375108 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.374969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c556e-6332-4919-b537-678db2c67d1d-istio-podinfo\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375108 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c556e-6332-4919-b537-678db2c67d1d-istiod-ca-cert\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375435 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-envoy\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375435 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-istio-token\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375435 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375239 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-certs\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375435 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375290 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-data\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375550 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-credential-socket\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375661 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375589 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-socket\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.375882 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.375858 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c556e-6332-4919-b537-678db2c67d1d-istiod-ca-cert\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.377404 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.377379 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-envoy\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.377623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.377604 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c556e-6332-4919-b537-678db2c67d1d-istio-podinfo\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.383427 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.383406 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-istio-token\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.383584 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.383550 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzd56\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-kube-api-access-hzd56\") pod \"e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.462319 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.462227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:08.608411 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.608388 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg"] Jul 07 00:14:08.610059 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:14:08.610028 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6c556e_6332_4919_b537_678db2c67d1d.slice/crio-4a10b2510b0e439739aa72b7f204099d10d25c63056d6aaa8979e343f3763b50 WatchSource:0}: Error finding container 4a10b2510b0e439739aa72b7f204099d10d25c63056d6aaa8979e343f3763b50: Status 404 returned error can't find the container with id 4a10b2510b0e439739aa72b7f204099d10d25c63056d6aaa8979e343f3763b50 Jul 07 00:14:08.612231 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.612193 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:14:08.612301 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.612282 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:14:08.612349 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.612331 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:14:08.917366 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.917330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" event={"ID":"2a6c556e-6332-4919-b537-678db2c67d1d","Type":"ContainerStarted","Data":"327f7061523978701d68dc102b4adf9dcca8e361f3168765cfede9affdc317ab"} Jul 07 00:14:08.917561 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.917388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" event={"ID":"2a6c556e-6332-4919-b537-678db2c67d1d","Type":"ContainerStarted","Data":"4a10b2510b0e439739aa72b7f204099d10d25c63056d6aaa8979e343f3763b50"} Jul 07 00:14:08.939343 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:08.939294 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podStartSLOduration=0.939276319 podStartE2EDuration="939.276319ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:14:08.938161874 +0000 UTC m=+1878.965067105" watchObservedRunningTime="2026-07-07 00:14:08.939276319 +0000 UTC m=+1878.966181549" Jul 07 00:14:09.463257 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:09.463212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:09.468241 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:09.468215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:09.925505 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:09.925466 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:09.926877 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:09.926851 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:10.909744 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:10.909713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:11.005670 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.005616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:11.071242 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.071199 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk"] Jul 07 00:14:11.071858 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.071836 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" Jul 07 00:14:11.072048 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.072026 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d12ae2-5d3a-498a-b763-0d005d766ebf" containerName="istio-proxy" Jul 07 00:14:11.075876 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.075857 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.078602 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.078580 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-0a7418-serving-cert\"" Jul 07 00:14:11.078765 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.078688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:14:11.086922 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.086897 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk"] Jul 07 00:14:11.107387 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.107353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:11.203361 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.203267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjn7t\" (UniqueName: \"kubernetes.io/projected/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-kube-api-access-qjn7t\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.203361 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.203314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.203361 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.203335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.255571 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.255536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:11.304583 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.304550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjn7t\" (UniqueName: \"kubernetes.io/projected/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-kube-api-access-qjn7t\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.304788 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.304592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.304788 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.304610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.305383 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.305360 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.307299 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.307278 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.319069 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.319037 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjn7t\" (UniqueName: \"kubernetes.io/projected/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-kube-api-access-qjn7t\") pod \"maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.392200 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.392163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:11.648559 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.648522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:11.738911 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.738888 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk"] Jul 07 00:14:11.740952 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:14:11.740916 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03de0c6a_3bb9_4e12_b28c_fad297c22aeb.slice/crio-1f9b802f4d7296fb2b7458b3ea966211fa678e657c121f643e3100a334920aa9 WatchSource:0}: Error finding container 1f9b802f4d7296fb2b7458b3ea966211fa678e657c121f643e3100a334920aa9: Status 404 returned error can't find the container with id 1f9b802f4d7296fb2b7458b3ea966211fa678e657c121f643e3100a334920aa9 Jul 07 00:14:11.742505 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.742473 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:14:11.743084 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.743055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:11.937778 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:11.937741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" event={"ID":"03de0c6a-3bb9-4e12-b28c-fad297c22aeb","Type":"ContainerStarted","Data":"1f9b802f4d7296fb2b7458b3ea966211fa678e657c121f643e3100a334920aa9"} Jul 07 00:14:12.944326 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:12.944245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" event={"ID":"03de0c6a-3bb9-4e12-b28c-fad297c22aeb","Type":"ContainerStarted","Data":"7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd"} Jul 07 00:14:12.944704 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:12.944322 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:12.963979 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:12.963930 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" podStartSLOduration=1.4324747119999999 podStartE2EDuration="1.963914433s" totalImagesPullingTime="531.439721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:14:12.961503216 +0000 UTC m=+1882.988408445" watchObservedRunningTime="2026-07-07 00:14:12.963914433 +0000 UTC m=+1882.990819668" Jul 07 00:14:15.437334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:15.437299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:15.790306 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:15.790271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:16.093242 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:16.093138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:16.344117 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:16.344026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:18.936728 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:18.936692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:18.956566 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:18.956535 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:14:18.983449 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:18.983418 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g"] Jul 07 00:14:18.987598 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:18.987571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:18.990142 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:18.990115 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-9caaee-serving-cert\"" Jul 07 00:14:19.000134 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.000094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g"] Jul 07 00:14:19.026074 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.026019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:19.076138 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.076098 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f15143c-c17f-4175-97b3-950e9e124606-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.076350 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.076156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7zn\" (UniqueName: \"kubernetes.io/projected/5f15143c-c17f-4175-97b3-950e9e124606-kube-api-access-rz7zn\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.076350 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.076308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f15143c-c17f-4175-97b3-950e9e124606-maas-api-tls\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.107343 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.107307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:19.177097 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.177066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f15143c-c17f-4175-97b3-950e9e124606-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.177284 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.177120 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7zn\" (UniqueName: \"kubernetes.io/projected/5f15143c-c17f-4175-97b3-950e9e124606-kube-api-access-rz7zn\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.177284 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.177165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f15143c-c17f-4175-97b3-950e9e124606-maas-api-tls\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.177728 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.177706 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f15143c-c17f-4175-97b3-950e9e124606-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.179568 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.179543 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f15143c-c17f-4175-97b3-950e9e124606-maas-api-tls\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.189877 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.189812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7zn\" (UniqueName: \"kubernetes.io/projected/5f15143c-c17f-4175-97b3-950e9e124606-kube-api-access-rz7zn\") pod \"maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.219483 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.219452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:19.318525 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.318492 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:19.461781 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.461753 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g"] Jul 07 00:14:19.463882 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:14:19.463855 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f15143c_c17f_4175_97b3_950e9e124606.slice/crio-eb7f90b707eef751039abfb89e7125137832de77c279f004308d8bee7c81f607 WatchSource:0}: Error finding container eb7f90b707eef751039abfb89e7125137832de77c279f004308d8bee7c81f607: Status 404 returned error can't find the container with id eb7f90b707eef751039abfb89e7125137832de77c279f004308d8bee7c81f607 Jul 07 00:14:19.604008 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.603973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:19.705318 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.705284 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:19.983071 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.983034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" event={"ID":"5f15143c-c17f-4175-97b3-950e9e124606","Type":"ContainerStarted","Data":"a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772"} Jul 07 00:14:19.983545 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.983093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" event={"ID":"5f15143c-c17f-4175-97b3-950e9e124606","Type":"ContainerStarted","Data":"eb7f90b707eef751039abfb89e7125137832de77c279f004308d8bee7c81f607"} Jul 07 00:14:19.983545 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:19.983116 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:20.001008 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:20.000959 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" podStartSLOduration=1.604407044 podStartE2EDuration="2.000943352s" totalImagesPullingTime="396.536308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:14:19.998556192 +0000 UTC m=+1890.025461432" watchObservedRunningTime="2026-07-07 00:14:20.000943352 +0000 UTC m=+1890.027848577" Jul 07 00:14:23.458070 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:23.458046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:25.967813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:25.967772 2564 patch_prober.go:28] interesting pod/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 00:14:25.968207 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:25.967834 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 00:14:25.993822 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:25.993790 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:14:27.021422 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:27.021325 2564 generic.go:356] "Generic (PLEG): container finished" podID="2a6c556e-6332-4919-b537-678db2c67d1d" containerID="327f7061523978701d68dc102b4adf9dcca8e361f3168765cfede9affdc317ab" exitCode=0 Jul 07 00:14:27.021422 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:27.021398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" event={"ID":"2a6c556e-6332-4919-b537-678db2c67d1d","Type":"ContainerDied","Data":"327f7061523978701d68dc102b4adf9dcca8e361f3168765cfede9affdc317ab"} Jul 07 00:14:27.021928 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:27.021818 2564 scope.go:117] "RemoveContainer" containerID="327f7061523978701d68dc102b4adf9dcca8e361f3168765cfede9affdc317ab" Jul 07 00:14:27.022830 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:27.022804 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:14:27.022906 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:27.022873 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:14:27.022906 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:27.022902 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:14:28.028557 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:28.028521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" event={"ID":"2a6c556e-6332-4919-b537-678db2c67d1d","Type":"ContainerStarted","Data":"a30d750910e500ed52fb2fdc25ae8db58a6cc64690ad992393bb2949d02be79a"} Jul 07 00:14:28.262958 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:28.262925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:28.462656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:28.462544 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:28.462916 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:28.462895 2564 patch_prober.go:28] interesting pod/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 07 00:14:28.462981 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:28.462954 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 07 00:14:29.462652 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:29.462595 2564 patch_prober.go:28] interesting pod/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 07 00:14:29.463039 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:29.462667 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 07 00:14:30.463194 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:30.463155 2564 patch_prober.go:28] interesting pod/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 07 00:14:30.463589 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:30.463209 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 07 00:14:31.463668 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:31.463602 2564 patch_prober.go:28] interesting pod/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 07 00:14:31.464150 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:31.463685 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 07 00:14:32.463555 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:32.463518 2564 patch_prober.go:28] interesting pod/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 07 00:14:32.463785 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:32.463578 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 07 00:14:32.515026 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:32.514984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:32.615001 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:32.614968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:33.466501 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:33.466472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:33.466939 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:33.466821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:33.467534 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:33.467513 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:14:40.324990 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:40.324952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:40.424794 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:40.424759 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:40.526105 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:40.526068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:46.422994 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.422956 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb"] Jul 07 00:14:46.427211 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.427191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.429662 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.429617 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9caaee\"/\"kube-root-ca.crt\"" Jul 07 00:14:46.430537 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.430498 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9caaee\"/\"openshift-service-ca.crt\"" Jul 07 00:14:46.430703 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.430527 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9caaee\"/\"test-model-9caaee-kserve-self-signed-certs\"" Jul 07 00:14:46.430703 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.430588 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9caaee\"/\"default-dockercfg-hz224\"" Jul 07 00:14:46.433753 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.433730 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb"] Jul 07 00:14:46.448159 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.448126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:14:46.534198 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.534154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbsdt\" (UniqueName: \"kubernetes.io/projected/a517eb29-951c-4b65-98a7-3596305e831c-kube-api-access-lbsdt\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.534385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.534228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-home\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.534385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.534265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a517eb29-951c-4b65-98a7-3596305e831c-tls-certs\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.534385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.534295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-tmp-dir\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.534385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.534321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-model-cache\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.534385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.534361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-dshm\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.635788 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.635744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-dshm\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636001 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.635793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbsdt\" (UniqueName: \"kubernetes.io/projected/a517eb29-951c-4b65-98a7-3596305e831c-kube-api-access-lbsdt\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636001 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.635859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-home\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636001 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.635886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a517eb29-951c-4b65-98a7-3596305e831c-tls-certs\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636180 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.636005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-tmp-dir\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636180 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.636046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-model-cache\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636288 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.636242 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-home\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636327 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.636302 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-tmp-dir\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.636419 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.636398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-model-cache\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.638219 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.638193 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-dshm\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.638389 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.638371 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a517eb29-951c-4b65-98a7-3596305e831c-tls-certs\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.644504 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.644483 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbsdt\" (UniqueName: \"kubernetes.io/projected/a517eb29-951c-4b65-98a7-3596305e831c-kube-api-access-lbsdt\") pod \"test-model-9caaee-kserve-77978577df-tngsb\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.739080 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.738995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:46.875402 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:46.875375 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb"] Jul 07 00:14:46.877470 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:14:46.877436 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda517eb29_951c_4b65_98a7_3596305e831c.slice/crio-333f3c823729fc643547342bed33157edda44ff2abf372c92808cc3bd35bd09d WatchSource:0}: Error finding container 333f3c823729fc643547342bed33157edda44ff2abf372c92808cc3bd35bd09d: Status 404 returned error can't find the container with id 333f3c823729fc643547342bed33157edda44ff2abf372c92808cc3bd35bd09d Jul 07 00:14:47.128372 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:47.128329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" event={"ID":"a517eb29-951c-4b65-98a7-3596305e831c","Type":"ContainerStarted","Data":"af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97"} Jul 07 00:14:47.128579 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:47.128382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" event={"ID":"a517eb29-951c-4b65-98a7-3596305e831c","Type":"ContainerStarted","Data":"333f3c823729fc643547342bed33157edda44ff2abf372c92808cc3bd35bd09d"} Jul 07 00:14:47.147266 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:47.147201 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" podStartSLOduration=1.147184025 podStartE2EDuration="1.147184025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:14:47.146538475 +0000 UTC m=+1917.173443705" watchObservedRunningTime="2026-07-07 00:14:47.147184025 +0000 UTC m=+1917.174089288" Jul 07 00:14:56.740101 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:56.740053 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:56.740101 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:56.740108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:56.752914 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:56.752885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:14:57.189303 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:14:57.189269 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:15:06.737269 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:06.737226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:07.661769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:07.661723 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb"] Jul 07 00:15:07.684647 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:07.684601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:07.831041 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:07.831004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:08.280930 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.280894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:08.384476 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.384439 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:08.483956 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.483917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:08.536738 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.536660 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk"] Jul 07 00:15:08.537003 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.536978 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" podUID="03de0c6a-3bb9-4e12-b28c-fad297c22aeb" containerName="maas-api" containerID="cri-o://7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd" gracePeriod=30 Jul 07 00:15:08.563842 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.563813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:08.789292 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.789227 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:15:08.935388 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.935201 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-openshift-service-ca\") pod \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " Jul 07 00:15:08.935388 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.935281 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjn7t\" (UniqueName: \"kubernetes.io/projected/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-kube-api-access-qjn7t\") pod \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " Jul 07 00:15:08.936084 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.935396 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-maas-api-tls\") pod \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\" (UID: \"03de0c6a-3bb9-4e12-b28c-fad297c22aeb\") " Jul 07 00:15:08.936084 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.935906 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03de0c6a-3bb9-4e12-b28c-fad297c22aeb" (UID: "03de0c6a-3bb9-4e12-b28c-fad297c22aeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:15:08.938062 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.938040 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-kube-api-access-qjn7t" (OuterVolumeSpecName: "kube-api-access-qjn7t") pod "03de0c6a-3bb9-4e12-b28c-fad297c22aeb" (UID: "03de0c6a-3bb9-4e12-b28c-fad297c22aeb"). InnerVolumeSpecName "kube-api-access-qjn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:15:08.938184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:08.938066 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03de0c6a-3bb9-4e12-b28c-fad297c22aeb" (UID: "03de0c6a-3bb9-4e12-b28c-fad297c22aeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:15:09.036840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.036800 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:09.036840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.036834 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjn7t\" (UniqueName: \"kubernetes.io/projected/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-kube-api-access-qjn7t\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:09.036840 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.036846 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03de0c6a-3bb9-4e12-b28c-fad297c22aeb-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:09.242812 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.242724 2564 generic.go:356] "Generic (PLEG): container finished" podID="03de0c6a-3bb9-4e12-b28c-fad297c22aeb" containerID="7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd" exitCode=0 Jul 07 00:15:09.242812 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.242773 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" Jul 07 00:15:09.242812 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.242797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" event={"ID":"03de0c6a-3bb9-4e12-b28c-fad297c22aeb","Type":"ContainerDied","Data":"7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd"} Jul 07 00:15:09.243098 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.242847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk" event={"ID":"03de0c6a-3bb9-4e12-b28c-fad297c22aeb","Type":"ContainerDied","Data":"1f9b802f4d7296fb2b7458b3ea966211fa678e657c121f643e3100a334920aa9"} Jul 07 00:15:09.243098 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.242868 2564 scope.go:117] "RemoveContainer" containerID="7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd" Jul 07 00:15:09.252574 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.252555 2564 scope.go:117] "RemoveContainer" containerID="7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd" Jul 07 00:15:09.252871 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:15:09.252852 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd\": container with ID starting with 7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd not found: ID does not exist" containerID="7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd" Jul 07 00:15:09.252935 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.252882 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd"} err="failed to get container status \"7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd\": rpc error: code = NotFound desc = could not find container \"7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd\": container with ID starting with 7d1ad9c9c42d6738e293c393d74bf4ae82b7eb8b729dd8fb418ff3b2600423dd not found: ID does not exist" Jul 07 00:15:09.265371 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.265347 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk"] Jul 07 00:15:09.269353 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:09.269327 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-0a7418-657b884bdc-5spqk"] Jul 07 00:15:10.654858 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:10.654828 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03de0c6a-3bb9-4e12-b28c-fad297c22aeb" path="/var/lib/kubelet/pods/03de0c6a-3bb9-4e12-b28c-fad297c22aeb/volumes" Jul 07 00:15:13.359953 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:13.359916 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:22.682504 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:22.682435 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" podUID="a517eb29-951c-4b65-98a7-3596305e831c" containerName="main" containerID="cri-o://af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97" gracePeriod=45 Jul 07 00:15:22.947006 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:22.946938 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:15:23.071277 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-dshm\") pod \"a517eb29-951c-4b65-98a7-3596305e831c\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " Jul 07 00:15:23.071474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071296 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbsdt\" (UniqueName: \"kubernetes.io/projected/a517eb29-951c-4b65-98a7-3596305e831c-kube-api-access-lbsdt\") pod \"a517eb29-951c-4b65-98a7-3596305e831c\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " Jul 07 00:15:23.071474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071365 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-tmp-dir\") pod \"a517eb29-951c-4b65-98a7-3596305e831c\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " Jul 07 00:15:23.071474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071397 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a517eb29-951c-4b65-98a7-3596305e831c-tls-certs\") pod \"a517eb29-951c-4b65-98a7-3596305e831c\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " Jul 07 00:15:23.071474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071430 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-model-cache\") pod \"a517eb29-951c-4b65-98a7-3596305e831c\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " Jul 07 00:15:23.071474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071463 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-home\") pod \"a517eb29-951c-4b65-98a7-3596305e831c\" (UID: \"a517eb29-951c-4b65-98a7-3596305e831c\") " Jul 07 00:15:23.071763 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071678 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a517eb29-951c-4b65-98a7-3596305e831c" (UID: "a517eb29-951c-4b65-98a7-3596305e831c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:23.071763 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071736 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a517eb29-951c-4b65-98a7-3596305e831c" (UID: "a517eb29-951c-4b65-98a7-3596305e831c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:23.071884 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.071767 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-home" (OuterVolumeSpecName: "home") pod "a517eb29-951c-4b65-98a7-3596305e831c" (UID: "a517eb29-951c-4b65-98a7-3596305e831c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:23.073592 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.073562 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a517eb29-951c-4b65-98a7-3596305e831c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a517eb29-951c-4b65-98a7-3596305e831c" (UID: "a517eb29-951c-4b65-98a7-3596305e831c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:15:23.073592 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.073587 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-dshm" (OuterVolumeSpecName: "dshm") pod "a517eb29-951c-4b65-98a7-3596305e831c" (UID: "a517eb29-951c-4b65-98a7-3596305e831c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:23.073778 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.073679 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a517eb29-951c-4b65-98a7-3596305e831c-kube-api-access-lbsdt" (OuterVolumeSpecName: "kube-api-access-lbsdt") pod "a517eb29-951c-4b65-98a7-3596305e831c" (UID: "a517eb29-951c-4b65-98a7-3596305e831c"). InnerVolumeSpecName "kube-api-access-lbsdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:15:23.172155 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.172111 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-tmp-dir\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:23.172155 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.172146 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a517eb29-951c-4b65-98a7-3596305e831c-tls-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:23.172155 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.172154 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-model-cache\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:23.172155 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.172163 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-home\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:23.172434 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.172172 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a517eb29-951c-4b65-98a7-3596305e831c-dshm\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:23.172434 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.172181 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbsdt\" (UniqueName: \"kubernetes.io/projected/a517eb29-951c-4b65-98a7-3596305e831c-kube-api-access-lbsdt\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:23.313927 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.313891 2564 generic.go:356] "Generic (PLEG): container finished" podID="a517eb29-951c-4b65-98a7-3596305e831c" containerID="af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97" exitCode=0 Jul 07 00:15:23.314132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.313939 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" Jul 07 00:15:23.314132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.313976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" event={"ID":"a517eb29-951c-4b65-98a7-3596305e831c","Type":"ContainerDied","Data":"af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97"} Jul 07 00:15:23.314132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.314025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb" event={"ID":"a517eb29-951c-4b65-98a7-3596305e831c","Type":"ContainerDied","Data":"333f3c823729fc643547342bed33157edda44ff2abf372c92808cc3bd35bd09d"} Jul 07 00:15:23.314132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.314048 2564 scope.go:117] "RemoveContainer" containerID="af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97" Jul 07 00:15:23.323246 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.323220 2564 scope.go:117] "RemoveContainer" containerID="af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97" Jul 07 00:15:23.323508 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:15:23.323488 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97\": container with ID starting with af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97 not found: ID does not exist" containerID="af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97" Jul 07 00:15:23.323559 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.323517 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97"} err="failed to get container status \"af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97\": rpc error: code = NotFound desc = could not find container \"af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97\": container with ID starting with af87b3682af8fce592003284f10d37c5ca6b78b52acd9b24fe9ebe950e271b97 not found: ID does not exist" Jul 07 00:15:23.336990 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.336958 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb"] Jul 07 00:15:23.340308 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:23.340286 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9caaee/test-model-9caaee-kserve-77978577df-tngsb"] Jul 07 00:15:24.654508 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:24.654470 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a517eb29-951c-4b65-98a7-3596305e831c" path="/var/lib/kubelet/pods/a517eb29-951c-4b65-98a7-3596305e831c/volumes" Jul 07 00:15:50.068491 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:50.068408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:50.342062 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:50.341977 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg"] Jul 07 00:15:50.342236 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:50.342197 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" containerID="cri-o://a30d750910e500ed52fb2fdc25ae8db58a6cc64690ad992393bb2949d02be79a" gracePeriod=30 Jul 07 00:15:50.369708 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:50.369556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:50.633503 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:50.633412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:50.805203 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:50.805162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:51.226439 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.226398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:51.325869 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.325826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:51.444991 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.444958 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g"] Jul 07 00:15:51.445288 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.445256 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" podUID="5f15143c-c17f-4175-97b3-950e9e124606" containerName="maas-api" containerID="cri-o://a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772" gracePeriod=30 Jul 07 00:15:51.476197 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.476172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:51.699920 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.699894 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:15:51.826364 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.826271 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f15143c-c17f-4175-97b3-950e9e124606-maas-api-tls\") pod \"5f15143c-c17f-4175-97b3-950e9e124606\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " Jul 07 00:15:51.826518 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.826376 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz7zn\" (UniqueName: \"kubernetes.io/projected/5f15143c-c17f-4175-97b3-950e9e124606-kube-api-access-rz7zn\") pod \"5f15143c-c17f-4175-97b3-950e9e124606\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " Jul 07 00:15:51.826518 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.826452 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f15143c-c17f-4175-97b3-950e9e124606-openshift-service-ca\") pod \"5f15143c-c17f-4175-97b3-950e9e124606\" (UID: \"5f15143c-c17f-4175-97b3-950e9e124606\") " Jul 07 00:15:51.826881 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.826854 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f15143c-c17f-4175-97b3-950e9e124606-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f15143c-c17f-4175-97b3-950e9e124606" (UID: "5f15143c-c17f-4175-97b3-950e9e124606"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:15:51.828544 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.828518 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f15143c-c17f-4175-97b3-950e9e124606-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f15143c-c17f-4175-97b3-950e9e124606" (UID: "5f15143c-c17f-4175-97b3-950e9e124606"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:15:51.828656 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.828546 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f15143c-c17f-4175-97b3-950e9e124606-kube-api-access-rz7zn" (OuterVolumeSpecName: "kube-api-access-rz7zn") pod "5f15143c-c17f-4175-97b3-950e9e124606" (UID: "5f15143c-c17f-4175-97b3-950e9e124606"). InnerVolumeSpecName "kube-api-access-rz7zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:15:51.927251 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.927204 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz7zn\" (UniqueName: \"kubernetes.io/projected/5f15143c-c17f-4175-97b3-950e9e124606-kube-api-access-rz7zn\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:51.927251 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.927246 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f15143c-c17f-4175-97b3-950e9e124606-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:51.927251 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:51.927258 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f15143c-c17f-4175-97b3-950e9e124606-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:52.460913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.460816 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f15143c-c17f-4175-97b3-950e9e124606" containerID="a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772" exitCode=0 Jul 07 00:15:52.460913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.460865 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" Jul 07 00:15:52.461476 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.460909 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" event={"ID":"5f15143c-c17f-4175-97b3-950e9e124606","Type":"ContainerDied","Data":"a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772"} Jul 07 00:15:52.461476 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.460978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g" event={"ID":"5f15143c-c17f-4175-97b3-950e9e124606","Type":"ContainerDied","Data":"eb7f90b707eef751039abfb89e7125137832de77c279f004308d8bee7c81f607"} Jul 07 00:15:52.461476 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.461012 2564 scope.go:117] "RemoveContainer" containerID="a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772" Jul 07 00:15:52.470221 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.470203 2564 scope.go:117] "RemoveContainer" containerID="a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772" Jul 07 00:15:52.470490 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:15:52.470461 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772\": container with ID starting with a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772 not found: ID does not exist" containerID="a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772" Jul 07 00:15:52.470538 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.470500 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772"} err="failed to get container status \"a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772\": rpc error: code = NotFound desc = could not find container \"a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772\": container with ID starting with a8bfa8cd068391c3e512db8de7d15ce9603338fafa72f234ba709d3d4c74f772 not found: ID does not exist" Jul 07 00:15:52.481554 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.481528 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g"] Jul 07 00:15:52.485377 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.485356 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-9caaee-5b5c4d94d5-w8t9g"] Jul 07 00:15:52.654783 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:52.654748 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f15143c-c17f-4175-97b3-950e9e124606" path="/var/lib/kubelet/pods/5f15143c-c17f-4175-97b3-950e9e124606/volumes" Jul 07 00:15:55.477731 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.477699 2564 generic.go:356] "Generic (PLEG): container finished" podID="2a6c556e-6332-4919-b537-678db2c67d1d" containerID="a30d750910e500ed52fb2fdc25ae8db58a6cc64690ad992393bb2949d02be79a" exitCode=0 Jul 07 00:15:55.478098 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.477768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" event={"ID":"2a6c556e-6332-4919-b537-678db2c67d1d","Type":"ContainerDied","Data":"a30d750910e500ed52fb2fdc25ae8db58a6cc64690ad992393bb2949d02be79a"} Jul 07 00:15:55.478098 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.477834 2564 scope.go:117] "RemoveContainer" containerID="327f7061523978701d68dc102b4adf9dcca8e361f3168765cfede9affdc317ab" Jul 07 00:15:55.608953 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.608890 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:15:55.763932 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.763893 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-data\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.763932 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.763939 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-credential-socket\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.763968 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c556e-6332-4919-b537-678db2c67d1d-istiod-ca-cert\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.763989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-envoy\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-istio-token\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzd56\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-kube-api-access-hzd56\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764121 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-certs\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764178 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-socket\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764495 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c556e-6332-4919-b537-678db2c67d1d-istio-podinfo\") pod \"2a6c556e-6332-4919-b537-678db2c67d1d\" (UID: \"2a6c556e-6332-4919-b537-678db2c67d1d\") " Jul 07 00:15:55.764495 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764229 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:55.764495 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764389 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:55.764495 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764452 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:55.764495 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764464 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6c556e-6332-4919-b537-678db2c67d1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:15:55.764770 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764575 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.764770 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764595 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.764770 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764609 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.764770 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764624 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c556e-6332-4919-b537-678db2c67d1d-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.764897 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.764774 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:55.766372 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.766341 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-kube-api-access-hzd56" (OuterVolumeSpecName: "kube-api-access-hzd56") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "kube-api-access-hzd56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:15:55.766485 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.766427 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:15:55.766565 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.766548 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a6c556e-6332-4919-b537-678db2c67d1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:15:55.766565 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.766557 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a6c556e-6332-4919-b537-678db2c67d1d" (UID: "2a6c556e-6332-4919-b537-678db2c67d1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:15:55.865230 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.865145 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.865230 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.865173 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.865230 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.865184 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzd56\" (UniqueName: \"kubernetes.io/projected/2a6c556e-6332-4919-b537-678db2c67d1d-kube-api-access-hzd56\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.865230 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.865193 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c556e-6332-4919-b537-678db2c67d1d-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:55.865230 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:55.865202 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c556e-6332-4919-b537-678db2c67d1d-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:15:56.484086 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:56.484052 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" Jul 07 00:15:56.484601 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:56.484096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg" event={"ID":"2a6c556e-6332-4919-b537-678db2c67d1d","Type":"ContainerDied","Data":"4a10b2510b0e439739aa72b7f204099d10d25c63056d6aaa8979e343f3763b50"} Jul 07 00:15:56.484601 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:56.484155 2564 scope.go:117] "RemoveContainer" containerID="a30d750910e500ed52fb2fdc25ae8db58a6cc64690ad992393bb2949d02be79a" Jul 07 00:15:56.505790 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:56.505758 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg"] Jul 07 00:15:56.514863 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:56.511982 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0a7418-openshift-default-c4cc596f5-62qrg"] Jul 07 00:15:56.654415 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:56.654369 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" path="/var/lib/kubelet/pods/2a6c556e-6332-4919-b537-678db2c67d1d/volumes" Jul 07 00:15:57.206878 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:57.206845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:57.270420 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:57.270384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:57.493704 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:57.493589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:57.760024 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:57.759938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:58.142557 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.142528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:58.466586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.466499 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55"] Jul 07 00:15:58.466962 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.466943 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" Jul 07 00:15:58.467092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.466965 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" Jul 07 00:15:58.467092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.466977 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f15143c-c17f-4175-97b3-950e9e124606" containerName="maas-api" Jul 07 00:15:58.467092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.466984 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f15143c-c17f-4175-97b3-950e9e124606" containerName="maas-api" Jul 07 00:15:58.467092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467002 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a517eb29-951c-4b65-98a7-3596305e831c" containerName="main" Jul 07 00:15:58.467092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467010 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a517eb29-951c-4b65-98a7-3596305e831c" containerName="main" Jul 07 00:15:58.467092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03de0c6a-3bb9-4e12-b28c-fad297c22aeb" containerName="maas-api" Jul 07 00:15:58.467092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467029 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="03de0c6a-3bb9-4e12-b28c-fad297c22aeb" containerName="maas-api" Jul 07 00:15:58.467457 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467146 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" Jul 07 00:15:58.467457 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467160 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="03de0c6a-3bb9-4e12-b28c-fad297c22aeb" containerName="maas-api" Jul 07 00:15:58.467457 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467169 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f15143c-c17f-4175-97b3-950e9e124606" containerName="maas-api" Jul 07 00:15:58.467457 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467180 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" Jul 07 00:15:58.467457 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.467190 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a517eb29-951c-4b65-98a7-3596305e831c" containerName="main" Jul 07 00:15:58.471903 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.471878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.474372 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.474344 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:15:58.474515 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.474473 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-69954e-openshift-default-dockercfg-5kgm6\"" Jul 07 00:15:58.482306 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.482276 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55"] Jul 07 00:15:58.488984 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.488952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:58.589624 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-socket\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.589624 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efb205e-e136-47a2-9889-b80512ac6e94-istiod-ca-cert\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.590051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-certs\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.590051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-data\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.590051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp4lv\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-kube-api-access-sp4lv\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.590051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-credential-socket\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.590051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efb205e-e136-47a2-9889-b80512ac6e94-istio-podinfo\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.590051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-istio-token\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.590051 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.589864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-envoy\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.643225 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.643189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:15:58.690892 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.690853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-credential-socket\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.690892 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.690891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efb205e-e136-47a2-9889-b80512ac6e94-istio-podinfo\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.690919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-istio-token\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.690949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-envoy\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.690985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-socket\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efb205e-e136-47a2-9889-b80512ac6e94-istiod-ca-cert\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-certs\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-data\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp4lv\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-kube-api-access-sp4lv\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691510 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691352 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-credential-socket\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691600 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691559 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-socket\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691685 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691595 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-data\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.691685 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.691623 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-certs\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.692061 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.692036 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efb205e-e136-47a2-9889-b80512ac6e94-istiod-ca-cert\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.693352 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.693331 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-envoy\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.693485 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.693465 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efb205e-e136-47a2-9889-b80512ac6e94-istio-podinfo\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.699911 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.699888 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-istio-token\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.700013 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.699979 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp4lv\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-kube-api-access-sp4lv\") pod \"e2e-api-a-69954e-openshift-default-857689d447-6qv55\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.783228 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.783190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:58.921930 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.921904 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55"] Jul 07 00:15:58.924087 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:15:58.924061 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efb205e_e136_47a2_9889_b80512ac6e94.slice/crio-5aa4df444ff64ca0882cb74ba4dfa3738076a9cc39cd972c16fbc4b375e95e7b WatchSource:0}: Error finding container 5aa4df444ff64ca0882cb74ba4dfa3738076a9cc39cd972c16fbc4b375e95e7b: Status 404 returned error can't find the container with id 5aa4df444ff64ca0882cb74ba4dfa3738076a9cc39cd972c16fbc4b375e95e7b Jul 07 00:15:58.929210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.929176 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:15:58.929302 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.929245 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:15:58.929302 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:58.929277 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:15:59.503039 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:59.502997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" event={"ID":"8efb205e-e136-47a2-9889-b80512ac6e94","Type":"ContainerStarted","Data":"89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3"} Jul 07 00:15:59.503214 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:59.503060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" event={"ID":"8efb205e-e136-47a2-9889-b80512ac6e94","Type":"ContainerStarted","Data":"5aa4df444ff64ca0882cb74ba4dfa3738076a9cc39cd972c16fbc4b375e95e7b"} Jul 07 00:15:59.522095 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:59.522045 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" podStartSLOduration=1.522028432 podStartE2EDuration="1.522028432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:15:59.521292336 +0000 UTC m=+1989.548197580" watchObservedRunningTime="2026-07-07 00:15:59.522028432 +0000 UTC m=+1989.548933661" Jul 07 00:15:59.784243 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:59.784211 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:15:59.789965 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:15:59.789937 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:00.507703 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:00.507668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:00.508558 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:00.508537 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:01.552700 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.552663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:01.646254 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.646222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:01.775558 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.775522 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq"] Jul 07 00:16:01.775962 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.775947 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" Jul 07 00:16:01.775962 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.775963 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6c556e-6332-4919-b537-678db2c67d1d" containerName="istio-proxy" Jul 07 00:16:01.786537 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.786503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:01.789312 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.789013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-69954e-serving-cert\"" Jul 07 00:16:01.789312 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.789197 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:16:01.790218 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.790187 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq"] Jul 07 00:16:01.817021 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.816927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:01.923191 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.923155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkxw\" (UniqueName: \"kubernetes.io/projected/84bc62b1-43af-4983-b969-1d35b9848970-kube-api-access-ffkxw\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:01.923378 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.923225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bc62b1-43af-4983-b969-1d35b9848970-openshift-service-ca\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:01.923378 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:01.923305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.024786 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.024751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bc62b1-43af-4983-b969-1d35b9848970-openshift-service-ca\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.025009 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.024838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.025009 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.024880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkxw\" (UniqueName: \"kubernetes.io/projected/84bc62b1-43af-4983-b969-1d35b9848970-kube-api-access-ffkxw\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.025084 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:02.025008 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-69954e-serving-cert: secret "maas-api-e2e-api-a-69954e-serving-cert" not found Jul 07 00:16:02.025120 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:02.025096 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls podName:84bc62b1-43af-4983-b969-1d35b9848970 nodeName:}" failed. No retries permitted until 2026-07-07 00:16:02.525075369 +0000 UTC m=+1992.551980593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls") pod "maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" (UID: "84bc62b1-43af-4983-b969-1d35b9848970") : secret "maas-api-e2e-api-a-69954e-serving-cert" not found Jul 07 00:16:02.025491 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.025471 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bc62b1-43af-4983-b969-1d35b9848970-openshift-service-ca\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.035223 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.035185 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkxw\" (UniqueName: \"kubernetes.io/projected/84bc62b1-43af-4983-b969-1d35b9848970-kube-api-access-ffkxw\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.099508 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.099377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:02.299257 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.299209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:02.401303 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.401207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:02.529936 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.529898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.532339 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.532314 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls\") pod \"maas-api-e2e-api-a-69954e-69dcdf564c-gckbq\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.699855 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.699764 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:02.845132 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:02.845102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq"] Jul 07 00:16:02.847016 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:16:02.846991 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bc62b1_43af_4983_b969_1d35b9848970.slice/crio-50986f6bbeb4146a9269d5d3373fc9e55a872febfcee992d6131e5bce74e3242 WatchSource:0}: Error finding container 50986f6bbeb4146a9269d5d3373fc9e55a872febfcee992d6131e5bce74e3242: Status 404 returned error can't find the container with id 50986f6bbeb4146a9269d5d3373fc9e55a872febfcee992d6131e5bce74e3242 Jul 07 00:16:03.526891 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:03.526749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" event={"ID":"84bc62b1-43af-4983-b969-1d35b9848970","Type":"ContainerStarted","Data":"c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389"} Jul 07 00:16:03.526891 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:03.526823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" event={"ID":"84bc62b1-43af-4983-b969-1d35b9848970","Type":"ContainerStarted","Data":"50986f6bbeb4146a9269d5d3373fc9e55a872febfcee992d6131e5bce74e3242"} Jul 07 00:16:03.526891 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:03.526854 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:03.544204 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:03.544156 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" podStartSLOduration=2.144800413 podStartE2EDuration="2.544140853s" totalImagesPullingTime="399.34044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:16:03.542093131 +0000 UTC m=+1993.568998354" watchObservedRunningTime="2026-07-07 00:16:03.544140853 +0000 UTC m=+1993.571046111" Jul 07 00:16:06.131485 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:06.131448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:06.616747 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:06.616706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:06.966517 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:06.966414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:07.073974 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:07.073935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:09.543904 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.543717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:09.548600 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.548570 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:09.613850 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.613821 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8"] Jul 07 00:16:09.624195 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.624159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:09.651665 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.649999 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8"] Jul 07 00:16:09.651665 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.650132 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.654268 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.653977 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-b7c63b-serving-cert\"" Jul 07 00:16:09.700915 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.700880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:09.798694 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.798579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d13f23a-0f4f-4d27-94b7-33474530fad4-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.798694 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.798666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.798888 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.798753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txlb\" (UniqueName: \"kubernetes.io/projected/7d13f23a-0f4f-4d27-94b7-33474530fad4-kube-api-access-7txlb\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.899582 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.899541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txlb\" (UniqueName: \"kubernetes.io/projected/7d13f23a-0f4f-4d27-94b7-33474530fad4-kube-api-access-7txlb\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.899791 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.899594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d13f23a-0f4f-4d27-94b7-33474530fad4-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.899791 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.899661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.899791 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:09.899770 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-b7c63b-serving-cert: secret "maas-api-e2e-api-b-b7c63b-serving-cert" not found Jul 07 00:16:09.899927 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:09.899839 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls podName:7d13f23a-0f4f-4d27-94b7-33474530fad4 nodeName:}" failed. No retries permitted until 2026-07-07 00:16:10.39982308 +0000 UTC m=+2000.426728287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls") pod "maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" (UID: "7d13f23a-0f4f-4d27-94b7-33474530fad4") : secret "maas-api-e2e-api-b-b7c63b-serving-cert" not found Jul 07 00:16:09.900242 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.900223 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d13f23a-0f4f-4d27-94b7-33474530fad4-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:09.912219 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:09.912177 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txlb\" (UniqueName: \"kubernetes.io/projected/7d13f23a-0f4f-4d27-94b7-33474530fad4-kube-api-access-7txlb\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:10.213096 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:10.212994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:10.310163 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:10.310127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:10.403782 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:10.403741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:10.406347 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:10.406322 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls\") pod \"maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:10.410203 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:10.410173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:10.566072 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:10.566038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:10.769217 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:10.769180 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8"] Jul 07 00:16:10.770821 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:16:10.770795 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d13f23a_0f4f_4d27_94b7_33474530fad4.slice/crio-1648b7731e9295d37c9f1dff4d29cedc56e47bfee2aac08a6555d3a1ce349c31 WatchSource:0}: Error finding container 1648b7731e9295d37c9f1dff4d29cedc56e47bfee2aac08a6555d3a1ce349c31: Status 404 returned error can't find the container with id 1648b7731e9295d37c9f1dff4d29cedc56e47bfee2aac08a6555d3a1ce349c31 Jul 07 00:16:11.569371 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:11.569331 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" event={"ID":"7d13f23a-0f4f-4d27-94b7-33474530fad4","Type":"ContainerStarted","Data":"d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05"} Jul 07 00:16:11.569805 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:11.569394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" event={"ID":"7d13f23a-0f4f-4d27-94b7-33474530fad4","Type":"ContainerStarted","Data":"1648b7731e9295d37c9f1dff4d29cedc56e47bfee2aac08a6555d3a1ce349c31"} Jul 07 00:16:11.569805 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:11.569455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:11.591435 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:11.591381 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" podStartSLOduration=2.220281509 podStartE2EDuration="2.591365718s" totalImagesPullingTime="371.084209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:16:11.588987053 +0000 UTC m=+2001.615892286" watchObservedRunningTime="2026-07-07 00:16:11.591365718 +0000 UTC m=+2001.618270948" Jul 07 00:16:14.068857 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:14.068825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:15.509258 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.509222 2564 patch_prober.go:28] interesting pod/e2e-api-a-69954e-openshift-default-857689d447-6qv55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 07 00:16:15.509720 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.509287 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 07 00:16:15.592688 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.592657 2564 generic.go:356] "Generic (PLEG): container finished" podID="8efb205e-e136-47a2-9889-b80512ac6e94" containerID="89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3" exitCode=0 Jul 07 00:16:15.592891 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.592722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" event={"ID":"8efb205e-e136-47a2-9889-b80512ac6e94","Type":"ContainerDied","Data":"89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3"} Jul 07 00:16:15.593185 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.593168 2564 scope.go:117] "RemoveContainer" containerID="89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3" Jul 07 00:16:15.594296 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.594266 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:15.594355 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.594334 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:15.594403 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:15.594368 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:16.600235 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:16.600201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" event={"ID":"8efb205e-e136-47a2-9889-b80512ac6e94","Type":"ContainerStarted","Data":"e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615"} Jul 07 00:16:16.783723 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:16.783678 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:16.784010 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:16.783985 2564 patch_prober.go:28] interesting pod/e2e-api-a-69954e-openshift-default-857689d447-6qv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 07 00:16:16.784076 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:16.784055 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 07 00:16:17.579834 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:17.579808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:17.784436 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:17.784406 2564 patch_prober.go:28] interesting pod/e2e-api-a-69954e-openshift-default-857689d447-6qv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 07 00:16:17.784813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:17.784471 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 07 00:16:18.784540 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:18.784500 2564 patch_prober.go:28] interesting pod/e2e-api-a-69954e-openshift-default-857689d447-6qv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 07 00:16:18.784982 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:18.784564 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 07 00:16:19.784492 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:19.784458 2564 patch_prober.go:28] interesting pod/e2e-api-a-69954e-openshift-default-857689d447-6qv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 07 00:16:19.784706 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:19.784524 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 07 00:16:20.794861 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:20.794829 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:20.795346 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:20.795109 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:20.795848 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:20.795830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:22.312657 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:22.312609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:22.712670 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:22.712561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:22.812540 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:22.812503 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:22.913129 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:22.913087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:23.427901 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.427867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:23.529043 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.529011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:23.646017 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.645979 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq"] Jul 07 00:16:23.646275 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.646220 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" podUID="84bc62b1-43af-4983-b969-1d35b9848970" containerName="maas-api" containerID="cri-o://c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389" gracePeriod=30 Jul 07 00:16:23.667684 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.667658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:23.896270 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.896245 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:23.929369 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.929335 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls\") pod \"84bc62b1-43af-4983-b969-1d35b9848970\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " Jul 07 00:16:23.929532 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.929374 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffkxw\" (UniqueName: \"kubernetes.io/projected/84bc62b1-43af-4983-b969-1d35b9848970-kube-api-access-ffkxw\") pod \"84bc62b1-43af-4983-b969-1d35b9848970\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " Jul 07 00:16:23.929532 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.929459 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bc62b1-43af-4983-b969-1d35b9848970-openshift-service-ca\") pod \"84bc62b1-43af-4983-b969-1d35b9848970\" (UID: \"84bc62b1-43af-4983-b969-1d35b9848970\") " Jul 07 00:16:23.929861 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.929837 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bc62b1-43af-4983-b969-1d35b9848970-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84bc62b1-43af-4983-b969-1d35b9848970" (UID: "84bc62b1-43af-4983-b969-1d35b9848970"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:16:23.931593 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.931562 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84bc62b1-43af-4983-b969-1d35b9848970" (UID: "84bc62b1-43af-4983-b969-1d35b9848970"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:16:23.931729 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:23.931658 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bc62b1-43af-4983-b969-1d35b9848970-kube-api-access-ffkxw" (OuterVolumeSpecName: "kube-api-access-ffkxw") pod "84bc62b1-43af-4983-b969-1d35b9848970" (UID: "84bc62b1-43af-4983-b969-1d35b9848970"). InnerVolumeSpecName "kube-api-access-ffkxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:16:24.030775 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.030743 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bc62b1-43af-4983-b969-1d35b9848970-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:24.030775 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.030774 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84bc62b1-43af-4983-b969-1d35b9848970-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:24.030971 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.030785 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffkxw\" (UniqueName: \"kubernetes.io/projected/84bc62b1-43af-4983-b969-1d35b9848970-kube-api-access-ffkxw\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:24.655058 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.655019 2564 generic.go:356] "Generic (PLEG): container finished" podID="84bc62b1-43af-4983-b969-1d35b9848970" containerID="c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389" exitCode=0 Jul 07 00:16:24.655545 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.655092 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" Jul 07 00:16:24.655545 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.655126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" event={"ID":"84bc62b1-43af-4983-b969-1d35b9848970","Type":"ContainerDied","Data":"c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389"} Jul 07 00:16:24.655545 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.655180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq" event={"ID":"84bc62b1-43af-4983-b969-1d35b9848970","Type":"ContainerDied","Data":"50986f6bbeb4146a9269d5d3373fc9e55a872febfcee992d6131e5bce74e3242"} Jul 07 00:16:24.655545 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.655215 2564 scope.go:117] "RemoveContainer" containerID="c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389" Jul 07 00:16:24.665739 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.665716 2564 scope.go:117] "RemoveContainer" containerID="c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389" Jul 07 00:16:24.665994 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:24.665976 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389\": container with ID starting with c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389 not found: ID does not exist" containerID="c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389" Jul 07 00:16:24.666050 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.666004 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389"} err="failed to get container status \"c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389\": rpc error: code = NotFound desc = could not find container \"c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389\": container with ID starting with c448602ef69d61cf95c34fdb05374edd42000db4952637e366a6d1318cdc3389 not found: ID does not exist" Jul 07 00:16:24.681542 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.681507 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq"] Jul 07 00:16:24.685186 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:24.685163 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-69954e-69dcdf564c-gckbq"] Jul 07 00:16:26.654769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:26.654734 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84bc62b1-43af-4983-b969-1d35b9848970" path="/var/lib/kubelet/pods/84bc62b1-43af-4983-b969-1d35b9848970/volumes" Jul 07 00:16:28.486456 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:28.486419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:28.581806 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:28.581766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:28.721170 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:28.721134 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55"] Jul 07 00:16:28.721424 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:28.721390 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" containerID="cri-o://e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615" gracePeriod=30 Jul 07 00:16:28.739784 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:28.739713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:28.983257 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:28.983217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:29.187568 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:29.187525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:29.626759 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:29.626727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:29.805115 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:29.805085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:29.902471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:29.902385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:30.017941 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.017907 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8"] Jul 07 00:16:30.018170 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.018140 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" podUID="7d13f23a-0f4f-4d27-94b7-33474530fad4" containerName="maas-api" containerID="cri-o://d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05" gracePeriod=30 Jul 07 00:16:30.043909 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.043885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:30.272613 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.272587 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:30.390227 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.390192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d13f23a-0f4f-4d27-94b7-33474530fad4-openshift-service-ca\") pod \"7d13f23a-0f4f-4d27-94b7-33474530fad4\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " Jul 07 00:16:30.390400 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.390264 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls\") pod \"7d13f23a-0f4f-4d27-94b7-33474530fad4\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " Jul 07 00:16:30.390400 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.390317 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7txlb\" (UniqueName: \"kubernetes.io/projected/7d13f23a-0f4f-4d27-94b7-33474530fad4-kube-api-access-7txlb\") pod \"7d13f23a-0f4f-4d27-94b7-33474530fad4\" (UID: \"7d13f23a-0f4f-4d27-94b7-33474530fad4\") " Jul 07 00:16:30.390606 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.390586 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d13f23a-0f4f-4d27-94b7-33474530fad4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d13f23a-0f4f-4d27-94b7-33474530fad4" (UID: "7d13f23a-0f4f-4d27-94b7-33474530fad4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:16:30.392453 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.392424 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d13f23a-0f4f-4d27-94b7-33474530fad4-kube-api-access-7txlb" (OuterVolumeSpecName: "kube-api-access-7txlb") pod "7d13f23a-0f4f-4d27-94b7-33474530fad4" (UID: "7d13f23a-0f4f-4d27-94b7-33474530fad4"). InnerVolumeSpecName "kube-api-access-7txlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:16:30.392560 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.392459 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d13f23a-0f4f-4d27-94b7-33474530fad4" (UID: "7d13f23a-0f4f-4d27-94b7-33474530fad4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:16:30.491750 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.491656 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txlb\" (UniqueName: \"kubernetes.io/projected/7d13f23a-0f4f-4d27-94b7-33474530fad4-kube-api-access-7txlb\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:30.491750 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.491689 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d13f23a-0f4f-4d27-94b7-33474530fad4-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:30.491750 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.491700 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d13f23a-0f4f-4d27-94b7-33474530fad4-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:30.687980 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.687943 2564 generic.go:356] "Generic (PLEG): container finished" podID="7d13f23a-0f4f-4d27-94b7-33474530fad4" containerID="d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05" exitCode=0 Jul 07 00:16:30.688471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.687993 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" Jul 07 00:16:30.688471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.688035 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" event={"ID":"7d13f23a-0f4f-4d27-94b7-33474530fad4","Type":"ContainerDied","Data":"d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05"} Jul 07 00:16:30.688471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.688102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8" event={"ID":"7d13f23a-0f4f-4d27-94b7-33474530fad4","Type":"ContainerDied","Data":"1648b7731e9295d37c9f1dff4d29cedc56e47bfee2aac08a6555d3a1ce349c31"} Jul 07 00:16:30.688471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.688127 2564 scope.go:117] "RemoveContainer" containerID="d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05" Jul 07 00:16:30.698184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.698163 2564 scope.go:117] "RemoveContainer" containerID="d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05" Jul 07 00:16:30.698485 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:30.698461 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05\": container with ID starting with d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05 not found: ID does not exist" containerID="d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05" Jul 07 00:16:30.698561 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.698496 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05"} err="failed to get container status \"d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05\": rpc error: code = NotFound desc = could not find container \"d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05\": container with ID starting with d669d640a4917c5971e4eb79b996b61ce1ce20d44eca5ccb3537cf46726e7f05 not found: ID does not exist" Jul 07 00:16:30.703953 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.703920 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8"] Jul 07 00:16:30.705681 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:30.705663 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7c63b-8db869fb7-nnrr8"] Jul 07 00:16:32.654729 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:32.654690 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d13f23a-0f4f-4d27-94b7-33474530fad4" path="/var/lib/kubelet/pods/7d13f23a-0f4f-4d27-94b7-33474530fad4/volumes" Jul 07 00:16:33.975439 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:33.975414 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:34.023649 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.023555 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-data\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.023649 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.023607 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-socket\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.023889 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.023654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-istio-token\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.023889 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.023672 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-certs\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.023889 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.023697 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp4lv\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-kube-api-access-sp4lv\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.024053 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.023901 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:16:34.024053 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.023956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efb205e-e136-47a2-9889-b80512ac6e94-istiod-ca-cert\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.024053 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024011 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efb205e-e136-47a2-9889-b80512ac6e94-istio-podinfo\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.024183 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024042 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:16:34.024183 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024059 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-envoy\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.024183 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-credential-socket\") pod \"8efb205e-e136-47a2-9889-b80512ac6e94\" (UID: \"8efb205e-e136-47a2-9889-b80512ac6e94\") " Jul 07 00:16:34.024415 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024394 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.024519 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024417 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.024519 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024418 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:16:34.024623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024549 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-data" (OuterVolumeSpecName: "istio-data") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:16:34.024791 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.024763 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efb205e-e136-47a2-9889-b80512ac6e94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:16:34.026171 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.026145 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:16:34.026431 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.026413 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-istio-token" (OuterVolumeSpecName: "istio-token") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:16:34.026534 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.026436 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8efb205e-e136-47a2-9889-b80512ac6e94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:16:34.026604 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.026553 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-kube-api-access-sp4lv" (OuterVolumeSpecName: "kube-api-access-sp4lv") pod "8efb205e-e136-47a2-9889-b80512ac6e94" (UID: "8efb205e-e136-47a2-9889-b80512ac6e94"). InnerVolumeSpecName "kube-api-access-sp4lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:16:34.125897 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.125857 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.125897 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.125890 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.125897 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.125899 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efb205e-e136-47a2-9889-b80512ac6e94-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.126129 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.125908 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.126129 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.125918 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp4lv\" (UniqueName: \"kubernetes.io/projected/8efb205e-e136-47a2-9889-b80512ac6e94-kube-api-access-sp4lv\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.126129 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.125926 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efb205e-e136-47a2-9889-b80512ac6e94-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.126129 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.125935 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efb205e-e136-47a2-9889-b80512ac6e94-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:16:34.708986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.708947 2564 generic.go:356] "Generic (PLEG): container finished" podID="8efb205e-e136-47a2-9889-b80512ac6e94" containerID="e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615" exitCode=0 Jul 07 00:16:34.709184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.708995 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" Jul 07 00:16:34.709184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.709027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" event={"ID":"8efb205e-e136-47a2-9889-b80512ac6e94","Type":"ContainerDied","Data":"e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615"} Jul 07 00:16:34.709184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.709095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55" event={"ID":"8efb205e-e136-47a2-9889-b80512ac6e94","Type":"ContainerDied","Data":"5aa4df444ff64ca0882cb74ba4dfa3738076a9cc39cd972c16fbc4b375e95e7b"} Jul 07 00:16:34.709184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.709118 2564 scope.go:117] "RemoveContainer" containerID="e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615" Jul 07 00:16:34.718225 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.718203 2564 scope.go:117] "RemoveContainer" containerID="89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3" Jul 07 00:16:34.726215 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.726197 2564 scope.go:117] "RemoveContainer" containerID="e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615" Jul 07 00:16:34.726532 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:34.726487 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615\": container with ID starting with e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615 not found: ID does not exist" containerID="e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615" Jul 07 00:16:34.726581 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.726542 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615"} err="failed to get container status \"e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615\": rpc error: code = NotFound desc = could not find container \"e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615\": container with ID starting with e28b84f6707d1ace99715c5fddfaf4a747b1290c88ad594ef50b5569f2efe615 not found: ID does not exist" Jul 07 00:16:34.726581 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.726560 2564 scope.go:117] "RemoveContainer" containerID="89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3" Jul 07 00:16:34.726869 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:34.726850 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3\": container with ID starting with 89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3 not found: ID does not exist" containerID="89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3" Jul 07 00:16:34.726928 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.726874 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3"} err="failed to get container status \"89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3\": rpc error: code = NotFound desc = could not find container \"89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3\": container with ID starting with 89e1363c5cb2d215f667fc85e746b424914434fd8916d930820ec4f963aa8ff3 not found: ID does not exist" Jul 07 00:16:34.734157 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.734135 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55"] Jul 07 00:16:34.750439 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:34.750410 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-69954e-openshift-default-857689d447-6qv55"] Jul 07 00:16:35.465547 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:35.465512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:35.537567 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:35.537533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:35.739407 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:35.739318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:35.966742 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:35.966710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:36.312295 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.312263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:36.609034 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.608957 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6"] Jul 07 00:16:36.609370 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609338 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" Jul 07 00:16:36.609370 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609350 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" Jul 07 00:16:36.609370 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609361 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" Jul 07 00:16:36.609370 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609366 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609376 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84bc62b1-43af-4983-b969-1d35b9848970" containerName="maas-api" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609384 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84bc62b1-43af-4983-b969-1d35b9848970" containerName="maas-api" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609399 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d13f23a-0f4f-4d27-94b7-33474530fad4" containerName="maas-api" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609404 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d13f23a-0f4f-4d27-94b7-33474530fad4" containerName="maas-api" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609480 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609487 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84bc62b1-43af-4983-b969-1d35b9848970" containerName="maas-api" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609493 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" containerName="istio-proxy" Jul 07 00:16:36.609507 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.609500 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d13f23a-0f4f-4d27-94b7-33474530fad4" containerName="maas-api" Jul 07 00:16:36.614161 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.614141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.616568 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.616547 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:16:36.616723 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.616581 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-72e474-openshift-default-dockercfg-nqgcj\"" Jul 07 00:16:36.627380 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.627359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6"] Jul 07 00:16:36.632040 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.632006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:36.656193 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.656159 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8efb205e-e136-47a2-9889-b80512ac6e94" path="/var/lib/kubelet/pods/8efb205e-e136-47a2-9889-b80512ac6e94/volumes" Jul 07 00:16:36.747871 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.747831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c28400f-af3c-4638-97cd-9efe461360c0-istio-podinfo\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748110 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.747885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-data\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748110 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.747975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-envoy\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748110 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.748040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-socket\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748110 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.748062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-istio-token\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748110 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.748104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-credential-socket\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748336 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.748186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-certs\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748336 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.748221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c28400f-af3c-4638-97cd-9efe461360c0-istiod-ca-cert\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.748336 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.748259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsb6n\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-kube-api-access-rsb6n\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.849586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-envoy\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.849813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-socket\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.849813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-istio-token\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.849813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-credential-socket\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.849813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-certs\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.849813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c28400f-af3c-4638-97cd-9efe461360c0-istiod-ca-cert\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.849813 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsb6n\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-kube-api-access-rsb6n\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.850135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c28400f-af3c-4638-97cd-9efe461360c0-istio-podinfo\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.850135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.849865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-data\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.850249 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.850163 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-certs\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.850249 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.850189 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-credential-socket\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.850363 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.850267 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-socket\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.850363 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.850338 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-data\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.850436 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.850421 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c28400f-af3c-4638-97cd-9efe461360c0-istiod-ca-cert\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.852020 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.851983 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-envoy\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.852497 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.852471 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c28400f-af3c-4638-97cd-9efe461360c0-istio-podinfo\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.852807 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.852783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:36.857142 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.857122 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsb6n\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-kube-api-access-rsb6n\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.857211 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.857154 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-istio-token\") pod \"e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:36.925744 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:36.925652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:37.070102 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.070074 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6"] Jul 07 00:16:37.072816 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:16:37.072781 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c28400f_af3c_4638_97cd_9efe461360c0.slice/crio-40b714d1869299bb8e54f262894036d2c2e3dd270ee9f1448926a069774d2654 WatchSource:0}: Error finding container 40b714d1869299bb8e54f262894036d2c2e3dd270ee9f1448926a069774d2654: Status 404 returned error can't find the container with id 40b714d1869299bb8e54f262894036d2c2e3dd270ee9f1448926a069774d2654 Jul 07 00:16:37.074922 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.074890 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:37.074986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.074959 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:37.075021 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.074992 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:37.728163 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.728119 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" event={"ID":"6c28400f-af3c-4638-97cd-9efe461360c0","Type":"ContainerStarted","Data":"8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177"} Jul 07 00:16:37.728674 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.728186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" event={"ID":"6c28400f-af3c-4638-97cd-9efe461360c0","Type":"ContainerStarted","Data":"40b714d1869299bb8e54f262894036d2c2e3dd270ee9f1448926a069774d2654"} Jul 07 00:16:37.750932 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.750879 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" podStartSLOduration=1.750862251 podStartE2EDuration="1.750862251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:16:37.748844916 +0000 UTC m=+2027.775750144" watchObservedRunningTime="2026-07-07 00:16:37.750862251 +0000 UTC m=+2027.777767479" Jul 07 00:16:37.926028 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:37.925990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:38.930911 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:38.930881 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:39.680623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.680580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:39.740973 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.740943 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:39.741257 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.741235 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4"] Jul 07 00:16:39.745304 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.745281 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:39.745550 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.745529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.748018 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.747996 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-26pnj\"" Jul 07 00:16:39.748334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.748311 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-72e474-serving-cert\"" Jul 07 00:16:39.754805 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.754781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4"] Jul 07 00:16:39.759799 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.759776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:39.832972 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.832930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:39.878614 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.878582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dae297a-229a-41fb-8376-731ff71c369f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.878614 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.878614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6wj\" (UniqueName: \"kubernetes.io/projected/7dae297a-229a-41fb-8376-731ff71c369f-kube-api-access-bj6wj\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.878866 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.878750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dae297a-229a-41fb-8376-731ff71c369f-maas-api-tls\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.933279 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.933196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:39.980070 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.980036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dae297a-229a-41fb-8376-731ff71c369f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.980070 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.980071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6wj\" (UniqueName: \"kubernetes.io/projected/7dae297a-229a-41fb-8376-731ff71c369f-kube-api-access-bj6wj\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.980323 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.980125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dae297a-229a-41fb-8376-731ff71c369f-maas-api-tls\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.980892 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.980861 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dae297a-229a-41fb-8376-731ff71c369f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.982689 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.982615 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dae297a-229a-41fb-8376-731ff71c369f-maas-api-tls\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:39.987966 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:39.987941 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6wj\" (UniqueName: \"kubernetes.io/projected/7dae297a-229a-41fb-8376-731ff71c369f-kube-api-access-bj6wj\") pod \"maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:40.060856 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.060816 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:40.211712 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.211685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4"] Jul 07 00:16:40.214271 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:16:40.214232 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dae297a_229a_41fb_8376_731ff71c369f.slice/crio-b34136f473f078fe82f4bf58bcd8b387dce2e5871fd31c1010aa300b42f803a8 WatchSource:0}: Error finding container b34136f473f078fe82f4bf58bcd8b387dce2e5871fd31c1010aa300b42f803a8: Status 404 returned error can't find the container with id b34136f473f078fe82f4bf58bcd8b387dce2e5871fd31c1010aa300b42f803a8 Jul 07 00:16:40.350984 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.350946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:40.451912 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.451877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:40.750194 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.750156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" event={"ID":"7dae297a-229a-41fb-8376-731ff71c369f","Type":"ContainerStarted","Data":"41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041"} Jul 07 00:16:40.750379 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.750212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" event={"ID":"7dae297a-229a-41fb-8376-731ff71c369f","Type":"ContainerStarted","Data":"b34136f473f078fe82f4bf58bcd8b387dce2e5871fd31c1010aa300b42f803a8"} Jul 07 00:16:40.750431 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.750418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:40.768807 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:40.768753 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" podStartSLOduration=1.364775534 podStartE2EDuration="1.768736509s" totalImagesPullingTime="403.960975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:16:40.766780527 +0000 UTC m=+2030.793685757" watchObservedRunningTime="2026-07-07 00:16:40.768736509 +0000 UTC m=+2030.795641737" Jul 07 00:16:44.250137 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:44.250103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:44.569928 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:44.569897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:44.889741 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:44.889648 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h"] Jul 07 00:16:44.893501 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:44.893476 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:44.896124 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:44.896103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-1ea658-openshift-default-dockercfg-f74vn\"" Jul 07 00:16:44.906966 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:44.906943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h"] Jul 07 00:16:44.923986 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:44.923950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:45.024423 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.024390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:45.027544 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-data\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.027700 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-credential-socket\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.027700 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istiod-ca-cert\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.027847 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-token\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.027847 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-socket\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.027847 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-envoy\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.027847 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhhl\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-kube-api-access-twhhl\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.028019 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-certs\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.028019 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.027905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-podinfo\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.128736 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-certs\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.128736 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-podinfo\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.128946 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-data\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.128946 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-credential-socket\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.128946 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istiod-ca-cert\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.128946 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-token\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.128946 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-socket\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.129198 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.128976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-envoy\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.129198 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.129002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhhl\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-kube-api-access-twhhl\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.129311 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.129229 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-data\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.129362 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.129342 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-credential-socket\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.129423 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.129377 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-socket\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.129423 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.129387 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-certs\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.129758 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.129734 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istiod-ca-cert\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.131210 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.131191 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-envoy\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.131329 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.131310 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-podinfo\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.137774 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.137753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-token\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.137953 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.137934 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhhl\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-kube-api-access-twhhl\") pod \"e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.205188 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.205104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:45.347433 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.347408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h"] Jul 07 00:16:45.349089 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:16:45.349062 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ae8531_4c41_4e80_a6ed_122bd4d9b05c.slice/crio-941e1d54ea3cec56a7de0ff13c4f747fb6ccc59f1fc26bb1f2780c1c389ca7f9 WatchSource:0}: Error finding container 941e1d54ea3cec56a7de0ff13c4f747fb6ccc59f1fc26bb1f2780c1c389ca7f9: Status 404 returned error can't find the container with id 941e1d54ea3cec56a7de0ff13c4f747fb6ccc59f1fc26bb1f2780c1c389ca7f9 Jul 07 00:16:45.351297 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.351263 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:45.351385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.351327 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:45.351385 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.351355 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:45.777124 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.777081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" event={"ID":"49ae8531-4c41-4e80-a6ed-122bd4d9b05c","Type":"ContainerStarted","Data":"82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e"} Jul 07 00:16:45.777310 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.777136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" event={"ID":"49ae8531-4c41-4e80-a6ed-122bd4d9b05c","Type":"ContainerStarted","Data":"941e1d54ea3cec56a7de0ff13c4f747fb6ccc59f1fc26bb1f2780c1c389ca7f9"} Jul 07 00:16:45.797412 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:45.797355 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" podStartSLOduration=1.797340977 podStartE2EDuration="1.797340977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:16:45.794887698 +0000 UTC m=+2035.821792938" watchObservedRunningTime="2026-07-07 00:16:45.797340977 +0000 UTC m=+2035.824246205" Jul 07 00:16:46.205327 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:46.205238 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:46.210915 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:46.210890 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:46.762046 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:46.762019 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:16:46.782674 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:46.782643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:46.783724 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:46.783701 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:16:47.696488 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.696444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:47.778533 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.778497 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v"] Jul 07 00:16:47.779280 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.779236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:47.782465 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.782443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.784856 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.784802 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-1ea658-serving-cert\"" Jul 07 00:16:47.790586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.790516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v"] Jul 07 00:16:47.855127 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.855093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618efe62-fdf8-4282-a10f-63992ecf7716-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.855304 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.855164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.855304 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.855193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4pz\" (UniqueName: \"kubernetes.io/projected/618efe62-fdf8-4282-a10f-63992ecf7716-kube-api-access-9h4pz\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.855653 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.855609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:47.956324 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.956217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.956324 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.956261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4pz\" (UniqueName: \"kubernetes.io/projected/618efe62-fdf8-4282-a10f-63992ecf7716-kube-api-access-9h4pz\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.956578 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:47.956387 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-1ea658-serving-cert: secret "maas-api-e2e-shared-b-1ea658-serving-cert" not found Jul 07 00:16:47.956578 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:16:47.956462 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls podName:618efe62-fdf8-4282-a10f-63992ecf7716 nodeName:}" failed. No retries permitted until 2026-07-07 00:16:48.456441725 +0000 UTC m=+2038.483346932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls") pod "maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" (UID: "618efe62-fdf8-4282-a10f-63992ecf7716") : secret "maas-api-e2e-shared-b-1ea658-serving-cert" not found Jul 07 00:16:47.956578 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.956516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618efe62-fdf8-4282-a10f-63992ecf7716-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.957181 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.957160 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618efe62-fdf8-4282-a10f-63992ecf7716-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:47.965272 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:47.965251 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4pz\" (UniqueName: \"kubernetes.io/projected/618efe62-fdf8-4282-a10f-63992ecf7716-kube-api-access-9h4pz\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:48.187414 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:48.187383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:48.287750 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:48.287715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:48.388055 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:48.388012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:48.461845 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:48.461803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:48.464224 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:48.464205 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls\") pod \"maas-api-e2e-shared-b-1ea658-bc688df75-crd4v\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:48.698691 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:48.698559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:48.907852 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:48.907824 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v"] Jul 07 00:16:48.909835 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:16:48.909813 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618efe62_fdf8_4282_a10f_63992ecf7716.slice/crio-38a16fa64c49805302af38bf3f8e375e7de11fbaa30c6390772173f529aafd0a WatchSource:0}: Error finding container 38a16fa64c49805302af38bf3f8e375e7de11fbaa30c6390772173f529aafd0a: Status 404 returned error can't find the container with id 38a16fa64c49805302af38bf3f8e375e7de11fbaa30c6390772173f529aafd0a Jul 07 00:16:49.805107 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:49.805067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" event={"ID":"618efe62-fdf8-4282-a10f-63992ecf7716","Type":"ContainerStarted","Data":"68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6"} Jul 07 00:16:49.805336 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:49.805130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" event={"ID":"618efe62-fdf8-4282-a10f-63992ecf7716","Type":"ContainerStarted","Data":"38a16fa64c49805302af38bf3f8e375e7de11fbaa30c6390772173f529aafd0a"} Jul 07 00:16:49.805336 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:49.805198 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:49.822652 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:49.822574 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" podStartSLOduration=2.476736433 podStartE2EDuration="2.822553771s" totalImagesPullingTime="345.817338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:16:49.820883887 +0000 UTC m=+2039.847789134" watchObservedRunningTime="2026-07-07 00:16:49.822553771 +0000 UTC m=+2039.849459024" Jul 07 00:16:52.201685 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:52.201611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:16:53.847360 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:53.847322 2564 generic.go:356] "Generic (PLEG): container finished" podID="6c28400f-af3c-4638-97cd-9efe461360c0" containerID="8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177" exitCode=0 Jul 07 00:16:53.847921 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:53.847398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" event={"ID":"6c28400f-af3c-4638-97cd-9efe461360c0","Type":"ContainerDied","Data":"8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177"} Jul 07 00:16:53.847998 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:53.847935 2564 scope.go:117] "RemoveContainer" containerID="8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177" Jul 07 00:16:53.848969 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:53.848942 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:53.849076 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:53.849022 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:53.849145 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:53.849072 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:54.741036 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:54.740989 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:54.859389 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:54.859336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" event={"ID":"6c28400f-af3c-4638-97cd-9efe461360c0","Type":"ContainerStarted","Data":"370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6"} Jul 07 00:16:54.926038 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:54.925999 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:54.927432 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:54.927402 2564 patch_prober.go:28] interesting pod/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 00:16:54.927682 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:54.927653 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 00:16:55.819738 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:55.819703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:16:55.926200 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:55.926163 2564 patch_prober.go:28] interesting pod/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 00:16:55.926807 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:55.926758 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 00:16:56.926974 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:56.926939 2564 patch_prober.go:28] interesting pod/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 00:16:56.927969 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:56.927546 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 00:16:57.926405 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:57.926365 2564 patch_prober.go:28] interesting pod/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 00:16:57.927160 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:57.926709 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 00:16:58.930868 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:58.930840 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:58.932252 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:58.931201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:58.932252 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:58.931930 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:16:59.888777 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:59.888745 2564 generic.go:356] "Generic (PLEG): container finished" podID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerID="82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e" exitCode=0 Jul 07 00:16:59.888964 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:59.888813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" event={"ID":"49ae8531-4c41-4e80-a6ed-122bd4d9b05c","Type":"ContainerDied","Data":"82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e"} Jul 07 00:16:59.889367 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:59.889352 2564 scope.go:117] "RemoveContainer" containerID="82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e" Jul 07 00:16:59.890289 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:59.890262 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:59.890361 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:59.890348 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:16:59.890398 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:16:59.890385 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 00:17:00.313565 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:00.313531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:17:00.895121 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:00.895083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" event={"ID":"49ae8531-4c41-4e80-a6ed-122bd4d9b05c","Type":"ContainerStarted","Data":"63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0"} Jul 07 00:17:01.205602 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:01.205509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:17:01.206808 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:01.206783 2564 patch_prober.go:28] interesting pod/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 07 00:17:01.206922 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:01.206845 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 07 00:17:01.782849 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:01.782807 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:17:02.140134 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:02.140038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:17:02.205607 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:02.205566 2564 patch_prober.go:28] interesting pod/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 07 00:17:02.205806 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:02.205662 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 07 00:17:02.241109 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:02.241073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:17:03.206394 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:03.206357 2564 patch_prober.go:28] interesting pod/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 07 00:17:03.206855 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:03.206412 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 07 00:17:04.206207 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:04.206168 2564 patch_prober.go:28] interesting pod/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 07 00:17:04.206382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:04.206230 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 07 00:17:05.209485 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:05.209454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:17:05.210471 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:05.210447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:17:09.155961 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:09.155922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:17:10.605439 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:10.605402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:17:25.626898 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:25.626813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:17:50.605130 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:50.605094 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:17:50.609702 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:50.609677 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:17:50.612811 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:50.612791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:17:50.616491 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:17:50.616474 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:19:11.399415 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:19:11.399379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:19:12.241705 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:19:12.241662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:19:23.025258 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:19:23.025221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:19:24.984850 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:19:24.984814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:19:25.082121 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:19:25.082085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:01.241950 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:20:01.241912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:18.737103 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:20:18.737017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:36.515878 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:20:36.515842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:37.366151 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:20:37.366117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:54.070233 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:20:54.070193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:19.475288 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:21:19.475256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:24.159622 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:21:24.159587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:24.259279 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:21:24.259236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:47.392956 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:21:47.392862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:47.493135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:21:47.493097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:02.139090 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:02.139053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:02.195259 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:02.195222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:02.943198 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:02.943165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:03.046785 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:03.046739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:09.786209 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:09.786172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:09.907059 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:09.907026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:10.009451 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:10.009419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:10.734424 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:10.734390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.204921 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.204886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.368590 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.368549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.469338 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.469245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.571761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.571720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.697716 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.697686 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4"] Jul 07 00:22:12.697931 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.697904 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" podUID="7dae297a-229a-41fb-8376-731ff71c369f" containerName="maas-api" containerID="cri-o://41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041" gracePeriod=30 Jul 07 00:22:12.727873 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.727787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.948168 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:12.948139 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:22:13.019805 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.019715 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj6wj\" (UniqueName: \"kubernetes.io/projected/7dae297a-229a-41fb-8376-731ff71c369f-kube-api-access-bj6wj\") pod \"7dae297a-229a-41fb-8376-731ff71c369f\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " Jul 07 00:22:13.019805 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.019757 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dae297a-229a-41fb-8376-731ff71c369f-maas-api-tls\") pod \"7dae297a-229a-41fb-8376-731ff71c369f\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " Jul 07 00:22:13.019994 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.019879 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dae297a-229a-41fb-8376-731ff71c369f-openshift-service-ca\") pod \"7dae297a-229a-41fb-8376-731ff71c369f\" (UID: \"7dae297a-229a-41fb-8376-731ff71c369f\") " Jul 07 00:22:13.020233 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.020206 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dae297a-229a-41fb-8376-731ff71c369f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dae297a-229a-41fb-8376-731ff71c369f" (UID: "7dae297a-229a-41fb-8376-731ff71c369f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:22:13.021868 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.021848 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dae297a-229a-41fb-8376-731ff71c369f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dae297a-229a-41fb-8376-731ff71c369f" (UID: "7dae297a-229a-41fb-8376-731ff71c369f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:22:13.022035 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.022014 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dae297a-229a-41fb-8376-731ff71c369f-kube-api-access-bj6wj" (OuterVolumeSpecName: "kube-api-access-bj6wj") pod "7dae297a-229a-41fb-8376-731ff71c369f" (UID: "7dae297a-229a-41fb-8376-731ff71c369f"). InnerVolumeSpecName "kube-api-access-bj6wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:22:13.121493 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.121463 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dae297a-229a-41fb-8376-731ff71c369f-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:22:13.121493 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.121487 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj6wj\" (UniqueName: \"kubernetes.io/projected/7dae297a-229a-41fb-8376-731ff71c369f-kube-api-access-bj6wj\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:22:13.121493 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.121497 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dae297a-229a-41fb-8376-731ff71c369f-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:22:13.484473 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.484436 2564 generic.go:356] "Generic (PLEG): container finished" podID="7dae297a-229a-41fb-8376-731ff71c369f" containerID="41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041" exitCode=0 Jul 07 00:22:13.484473 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.484467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" event={"ID":"7dae297a-229a-41fb-8376-731ff71c369f","Type":"ContainerDied","Data":"41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041"} Jul 07 00:22:13.484970 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.484521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" event={"ID":"7dae297a-229a-41fb-8376-731ff71c369f","Type":"ContainerDied","Data":"b34136f473f078fe82f4bf58bcd8b387dce2e5871fd31c1010aa300b42f803a8"} Jul 07 00:22:13.484970 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.484544 2564 scope.go:117] "RemoveContainer" containerID="41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041" Jul 07 00:22:13.484970 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.484481 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4" Jul 07 00:22:13.494190 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.494169 2564 scope.go:117] "RemoveContainer" containerID="41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041" Jul 07 00:22:13.494468 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:22:13.494449 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041\": container with ID starting with 41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041 not found: ID does not exist" containerID="41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041" Jul 07 00:22:13.494528 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.494477 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041"} err="failed to get container status \"41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041\": rpc error: code = NotFound desc = could not find container \"41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041\": container with ID starting with 41c62e768919c447f1b78fca3cab61dc7bb9fab2b142c8680f6153a499c21041 not found: ID does not exist" Jul 07 00:22:13.506703 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.506674 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4"] Jul 07 00:22:13.509598 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:13.509578 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-72e474-bc7c98d6c-22gf4"] Jul 07 00:22:14.655684 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:14.655625 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dae297a-229a-41fb-8376-731ff71c369f" path="/var/lib/kubelet/pods/7dae297a-229a-41fb-8376-731ff71c369f/volumes" Jul 07 00:22:17.420171 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:17.420131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:50.636899 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:50.636869 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:22:50.640387 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:50.640366 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:22:50.647315 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:50.647282 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:22:50.651437 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:50.651416 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:22:54.545131 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:54.545095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:54.808695 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:54.808588 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6"] Jul 07 00:22:54.808880 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:54.808841 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" containerID="cri-o://370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6" gracePeriod=30 Jul 07 00:22:54.854119 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:54.854087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:55.114228 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:55.114141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:55.308015 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:55.307979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:55.722522 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:55.722493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:55.847267 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:55.847233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:55.974483 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:55.974392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:56.068018 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.067986 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v"] Jul 07 00:22:56.068293 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.068266 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" podUID="618efe62-fdf8-4282-a10f-63992ecf7716" containerName="maas-api" containerID="cri-o://68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6" gracePeriod=30 Jul 07 00:22:56.119925 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.119893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:56.323870 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.323839 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:22:56.404324 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.404290 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618efe62-fdf8-4282-a10f-63992ecf7716-openshift-service-ca\") pod \"618efe62-fdf8-4282-a10f-63992ecf7716\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " Jul 07 00:22:56.404509 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.404346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h4pz\" (UniqueName: \"kubernetes.io/projected/618efe62-fdf8-4282-a10f-63992ecf7716-kube-api-access-9h4pz\") pod \"618efe62-fdf8-4282-a10f-63992ecf7716\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " Jul 07 00:22:56.404509 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.404369 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls\") pod \"618efe62-fdf8-4282-a10f-63992ecf7716\" (UID: \"618efe62-fdf8-4282-a10f-63992ecf7716\") " Jul 07 00:22:56.404700 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.404676 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618efe62-fdf8-4282-a10f-63992ecf7716-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "618efe62-fdf8-4282-a10f-63992ecf7716" (UID: "618efe62-fdf8-4282-a10f-63992ecf7716"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:22:56.406476 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.406448 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "618efe62-fdf8-4282-a10f-63992ecf7716" (UID: "618efe62-fdf8-4282-a10f-63992ecf7716"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:22:56.406476 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.406458 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618efe62-fdf8-4282-a10f-63992ecf7716-kube-api-access-9h4pz" (OuterVolumeSpecName: "kube-api-access-9h4pz") pod "618efe62-fdf8-4282-a10f-63992ecf7716" (UID: "618efe62-fdf8-4282-a10f-63992ecf7716"). InnerVolumeSpecName "kube-api-access-9h4pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:22:56.506016 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.505922 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618efe62-fdf8-4282-a10f-63992ecf7716-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:22:56.506016 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.505955 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4pz\" (UniqueName: \"kubernetes.io/projected/618efe62-fdf8-4282-a10f-63992ecf7716-kube-api-access-9h4pz\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:22:56.506016 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.505965 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618efe62-fdf8-4282-a10f-63992ecf7716-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:22:56.699533 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.699505 2564 generic.go:356] "Generic (PLEG): container finished" podID="618efe62-fdf8-4282-a10f-63992ecf7716" containerID="68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6" exitCode=0 Jul 07 00:22:56.699714 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.699546 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" Jul 07 00:22:56.699714 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.699583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" event={"ID":"618efe62-fdf8-4282-a10f-63992ecf7716","Type":"ContainerDied","Data":"68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6"} Jul 07 00:22:56.699714 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.699647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v" event={"ID":"618efe62-fdf8-4282-a10f-63992ecf7716","Type":"ContainerDied","Data":"38a16fa64c49805302af38bf3f8e375e7de11fbaa30c6390772173f529aafd0a"} Jul 07 00:22:56.699714 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.699675 2564 scope.go:117] "RemoveContainer" containerID="68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6" Jul 07 00:22:56.709403 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.709386 2564 scope.go:117] "RemoveContainer" containerID="68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6" Jul 07 00:22:56.709697 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:22:56.709677 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6\": container with ID starting with 68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6 not found: ID does not exist" containerID="68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6" Jul 07 00:22:56.709772 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.709705 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6"} err="failed to get container status \"68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6\": rpc error: code = NotFound desc = could not find container \"68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6\": container with ID starting with 68fbc45f9d5583c6bda00a018460c9bf1b3122606964a141251ff3c73147dcc6 not found: ID does not exist" Jul 07 00:22:56.716539 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.716507 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v"] Jul 07 00:22:56.720697 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:56.720670 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-1ea658-bc688df75-crd4v"] Jul 07 00:22:58.654770 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:58.654737 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618efe62-fdf8-4282-a10f-63992ecf7716" path="/var/lib/kubelet/pods/618efe62-fdf8-4282-a10f-63992ecf7716/volumes" Jul 07 00:22:58.934177 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:58.934082 2564 patch_prober.go:28] interesting pod/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 00:22:58.934177 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:22:58.934160 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 00:23:00.059018 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.058995 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:23:00.139623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139533 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-credential-socket\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139570 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsb6n\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-kube-api-access-rsb6n\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139597 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-envoy\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-data\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139670 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-istio-token\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139735 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-socket\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c28400f-af3c-4638-97cd-9efe461360c0-istiod-ca-cert\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139819 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-certs\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.139913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139851 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:00.139913 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.139863 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c28400f-af3c-4638-97cd-9efe461360c0-istio-podinfo\") pod \"6c28400f-af3c-4638-97cd-9efe461360c0\" (UID: \"6c28400f-af3c-4638-97cd-9efe461360c0\") " Jul 07 00:23:00.140276 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.140231 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.140276 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.140251 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c28400f-af3c-4638-97cd-9efe461360c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:23:00.140383 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.140312 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:00.140463 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.140435 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:00.140770 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.140744 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:00.141984 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.141963 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:00.142080 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.142055 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c28400f-af3c-4638-97cd-9efe461360c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:23:00.142234 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.142212 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:23:00.142333 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.142230 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-kube-api-access-rsb6n" (OuterVolumeSpecName: "kube-api-access-rsb6n") pod "6c28400f-af3c-4638-97cd-9efe461360c0" (UID: "6c28400f-af3c-4638-97cd-9efe461360c0"). InnerVolumeSpecName "kube-api-access-rsb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:23:00.241043 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241003 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c28400f-af3c-4638-97cd-9efe461360c0-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.241043 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241037 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.241043 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241045 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c28400f-af3c-4638-97cd-9efe461360c0-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.241334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241055 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsb6n\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-kube-api-access-rsb6n\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.241334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241067 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.241334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241075 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.241334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241082 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c28400f-af3c-4638-97cd-9efe461360c0-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.241334 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.241089 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c28400f-af3c-4638-97cd-9efe461360c0-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:00.728000 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.727961 2564 generic.go:356] "Generic (PLEG): container finished" podID="6c28400f-af3c-4638-97cd-9efe461360c0" containerID="370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6" exitCode=0 Jul 07 00:23:00.728156 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.728025 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" Jul 07 00:23:00.728156 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.728049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" event={"ID":"6c28400f-af3c-4638-97cd-9efe461360c0","Type":"ContainerDied","Data":"370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6"} Jul 07 00:23:00.728156 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.728126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6" event={"ID":"6c28400f-af3c-4638-97cd-9efe461360c0","Type":"ContainerDied","Data":"40b714d1869299bb8e54f262894036d2c2e3dd270ee9f1448926a069774d2654"} Jul 07 00:23:00.728156 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.728150 2564 scope.go:117] "RemoveContainer" containerID="370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6" Jul 07 00:23:00.736828 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.736812 2564 scope.go:117] "RemoveContainer" containerID="8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177" Jul 07 00:23:00.744824 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.744793 2564 scope.go:117] "RemoveContainer" containerID="370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6" Jul 07 00:23:00.745299 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:23:00.745277 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6\": container with ID starting with 370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6 not found: ID does not exist" containerID="370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6" Jul 07 00:23:00.745401 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.745307 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6"} err="failed to get container status \"370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6\": rpc error: code = NotFound desc = could not find container \"370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6\": container with ID starting with 370466105abd817707529be9c75c986f969913d9c84d4c130178311b6679f4e6 not found: ID does not exist" Jul 07 00:23:00.745401 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.745332 2564 scope.go:117] "RemoveContainer" containerID="8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177" Jul 07 00:23:00.745593 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:23:00.745577 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177\": container with ID starting with 8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177 not found: ID does not exist" containerID="8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177" Jul 07 00:23:00.745674 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.745601 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177"} err="failed to get container status \"8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177\": rpc error: code = NotFound desc = could not find container \"8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177\": container with ID starting with 8e2125af190230049eb0a05598e5c79e0398e25c6c70784b65def01baa71c177 not found: ID does not exist" Jul 07 00:23:00.747212 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.747194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6"] Jul 07 00:23:00.753343 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:00.753323 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-72e474-openshift-default-84458c56b7-4zlc6"] Jul 07 00:23:01.871572 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:01.871539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:01.972574 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:01.972527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:02.176730 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:02.176642 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h"] Jul 07 00:23:02.176884 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:02.176863 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" containerID="cri-o://63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0" gracePeriod=30 Jul 07 00:23:02.214721 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:02.214685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:02.426184 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:02.426141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:02.654713 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:02.654672 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" path="/var/lib/kubelet/pods/6c28400f-af3c-4638-97cd-9efe461360c0/volumes" Jul 07 00:23:05.211958 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:05.211924 2564 patch_prober.go:28] interesting pod/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 00:23:05.212413 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:05.211979 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 00:23:07.430503 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.430478 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:23:07.506460 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twhhl\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-kube-api-access-twhhl\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506460 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506421 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-credential-socket\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506460 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506440 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-socket\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506475 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-podinfo\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506504 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istiod-ca-cert\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506534 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-data\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-envoy\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506656 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-token\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506691 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-certs\") pod \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\" (UID: \"49ae8531-4c41-4e80-a6ed-122bd4d9b05c\") " Jul 07 00:23:07.506761 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506750 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:07.507119 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506959 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.507119 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.506966 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:23:07.507227 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.507151 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:07.507227 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.507169 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:07.507423 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.507396 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-data" (OuterVolumeSpecName: "istio-data") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:07.508822 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.508791 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-kube-api-access-twhhl" (OuterVolumeSpecName: "kube-api-access-twhhl") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "kube-api-access-twhhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:23:07.508822 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.508815 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-token" (OuterVolumeSpecName: "istio-token") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:23:07.508955 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.508851 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:23:07.508955 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.508889 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49ae8531-4c41-4e80-a6ed-122bd4d9b05c" (UID: "49ae8531-4c41-4e80-a6ed-122bd4d9b05c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.607875 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.607939 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.607980 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.607997 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhhl\" (UniqueName: \"kubernetes.io/projected/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-kube-api-access-twhhl\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.608017 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.608041 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.608055 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.611666 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.608070 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ae8531-4c41-4e80-a6ed-122bd4d9b05c-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 07 00:23:07.766078 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.765985 2564 generic.go:356] "Generic (PLEG): container finished" podID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerID="63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0" exitCode=0 Jul 07 00:23:07.766078 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.766039 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" Jul 07 00:23:07.766314 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.766066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" event={"ID":"49ae8531-4c41-4e80-a6ed-122bd4d9b05c","Type":"ContainerDied","Data":"63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0"} Jul 07 00:23:07.766314 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.766132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h" event={"ID":"49ae8531-4c41-4e80-a6ed-122bd4d9b05c","Type":"ContainerDied","Data":"941e1d54ea3cec56a7de0ff13c4f747fb6ccc59f1fc26bb1f2780c1c389ca7f9"} Jul 07 00:23:07.766314 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.766165 2564 scope.go:117] "RemoveContainer" containerID="63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0" Jul 07 00:23:07.775285 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.775260 2564 scope.go:117] "RemoveContainer" containerID="82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e" Jul 07 00:23:07.783122 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.783107 2564 scope.go:117] "RemoveContainer" containerID="63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0" Jul 07 00:23:07.783361 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:23:07.783337 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0\": container with ID starting with 63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0 not found: ID does not exist" containerID="63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0" Jul 07 00:23:07.783398 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.783369 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0"} err="failed to get container status \"63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0\": rpc error: code = NotFound desc = could not find container \"63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0\": container with ID starting with 63ceae8d6f439b3225bce2df94f07e47dcaca09d78db7c59ff7449ba29173eb0 not found: ID does not exist" Jul 07 00:23:07.783398 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.783387 2564 scope.go:117] "RemoveContainer" containerID="82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e" Jul 07 00:23:07.783647 ip-10-0-132-183 kubenswrapper[2564]: E0707 00:23:07.783609 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e\": container with ID starting with 82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e not found: ID does not exist" containerID="82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e" Jul 07 00:23:07.783692 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.783659 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e"} err="failed to get container status \"82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e\": rpc error: code = NotFound desc = could not find container \"82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e\": container with ID starting with 82b5233a5baa6ed8c9c9558d7a041f4887a154cab96a1dcfba5f5de01e78e93e not found: ID does not exist" Jul 07 00:23:07.801998 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.801970 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h"] Jul 07 00:23:07.803658 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.803618 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-1ea658-openshift-default-6c9848cb4c-8vl9h"] Jul 07 00:23:07.849474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.849446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5ddd4945c8-52wgr_4cf12e6e-add2-4904-97a1-32e9f5db5af2/maas-api/0.log" Jul 07 00:23:07.962687 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:07.962657 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723040-94xpf_c971ef3f-1281-4d73-a14f-cdd896dff546/cleanup/0.log" Jul 07 00:23:08.213451 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:08.213411 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-696cfbbf9b-pm4gt_eaddac64-3801-478c-9737-f59ba2f8c621/manager/0.log" Jul 07 00:23:08.559027 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:08.559003 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6688846b76-hwrjr_a1510dbb-cae6-4353-8bad-e7ddda73a8ac/copy-manifests/0.log" Jul 07 00:23:08.579326 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:08.579294 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6688846b76-hwrjr_a1510dbb-cae6-4353-8bad-e7ddda73a8ac/manager/0.log" Jul 07 00:23:08.655215 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:08.655181 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" path="/var/lib/kubelet/pods/49ae8531-4c41-4e80-a6ed-122bd4d9b05c/volumes" Jul 07 00:23:08.935474 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:08.935393 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-rwzgz_c56466fa-f92c-42f2-9756-494d2003a9d6/postgres/0.log" Jul 07 00:23:10.405623 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:10.405590 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-r5t95_d1d94964-164e-449e-b637-2a683b6d6620/manager/0.log" Jul 07 00:23:10.760488 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:10.760409 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hdjb4_aa521a3e-2db8-4adb-bb07-ecba685763f3/registry-server/0.log" Jul 07 00:23:12.223763 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:12.223729 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lw5ck_92a1830b-dcd2-4e3d-b888-ef32b1f84442/payload-processing/0.log" Jul 07 00:23:12.341504 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:12.341475 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d86c67597-d9q6b_0522ae39-76b7-493c-9a01-2913bf880b08/router/0.log" Jul 07 00:23:12.811260 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:12.811184 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-t2fmt_ca189b38-4f40-4d5e-aaec-f63a5900b29a/main/0.log" Jul 07 00:23:13.043572 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:13.043541 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfdl8x_b4002879-478f-4b5d-8a40-fc324acc7e3d/main/0.log" Jul 07 00:23:17.547000 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.546967 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tmw4w/must-gather-n5x62"] Jul 07 00:23:17.547382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547327 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618efe62-fdf8-4282-a10f-63992ecf7716" containerName="maas-api" Jul 07 00:23:17.547382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547339 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="618efe62-fdf8-4282-a10f-63992ecf7716" containerName="maas-api" Jul 07 00:23:17.547382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547353 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" Jul 07 00:23:17.547382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547359 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" Jul 07 00:23:17.547382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547370 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dae297a-229a-41fb-8376-731ff71c369f" containerName="maas-api" Jul 07 00:23:17.547382 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547375 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dae297a-229a-41fb-8376-731ff71c369f" containerName="maas-api" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547387 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547392 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547404 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547411 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547486 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547493 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547500 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ae8531-4c41-4e80-a6ed-122bd4d9b05c" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547507 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dae297a-229a-41fb-8376-731ff71c369f" containerName="maas-api" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547515 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="618efe62-fdf8-4282-a10f-63992ecf7716" containerName="maas-api" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547580 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" Jul 07 00:23:17.547594 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547594 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" Jul 07 00:23:17.547947 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.547663 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c28400f-af3c-4638-97cd-9efe461360c0" containerName="istio-proxy" Jul 07 00:23:17.552073 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.552048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.556452 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.556421 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tmw4w\"/\"kube-root-ca.crt\"" Jul 07 00:23:17.556586 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.556451 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tmw4w\"/\"openshift-service-ca.crt\"" Jul 07 00:23:17.558725 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.558702 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tmw4w\"/\"default-dockercfg-gdztz\"" Jul 07 00:23:17.561399 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.561375 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmw4w/must-gather-n5x62"] Jul 07 00:23:17.564390 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.564365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:17.665143 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.665109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:17.698068 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.698037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38a6ae92-90b3-49c8-8300-84a8cb1cd68e-must-gather-output\") pod \"must-gather-n5x62\" (UID: \"38a6ae92-90b3-49c8-8300-84a8cb1cd68e\") " pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.698068 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.698068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xq6\" (UniqueName: \"kubernetes.io/projected/38a6ae92-90b3-49c8-8300-84a8cb1cd68e-kube-api-access-v6xq6\") pod \"must-gather-n5x62\" (UID: \"38a6ae92-90b3-49c8-8300-84a8cb1cd68e\") " pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.799181 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.799101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38a6ae92-90b3-49c8-8300-84a8cb1cd68e-must-gather-output\") pod \"must-gather-n5x62\" (UID: \"38a6ae92-90b3-49c8-8300-84a8cb1cd68e\") " pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.799181 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.799134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xq6\" (UniqueName: \"kubernetes.io/projected/38a6ae92-90b3-49c8-8300-84a8cb1cd68e-kube-api-access-v6xq6\") pod \"must-gather-n5x62\" (UID: \"38a6ae92-90b3-49c8-8300-84a8cb1cd68e\") " pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.799458 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.799440 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38a6ae92-90b3-49c8-8300-84a8cb1cd68e-must-gather-output\") pod \"must-gather-n5x62\" (UID: \"38a6ae92-90b3-49c8-8300-84a8cb1cd68e\") " pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.806934 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.806907 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xq6\" (UniqueName: \"kubernetes.io/projected/38a6ae92-90b3-49c8-8300-84a8cb1cd68e-kube-api-access-v6xq6\") pod \"must-gather-n5x62\" (UID: \"38a6ae92-90b3-49c8-8300-84a8cb1cd68e\") " pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.862068 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.862038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmw4w/must-gather-n5x62" Jul 07 00:23:17.994259 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.994236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmw4w/must-gather-n5x62"] Jul 07 00:23:17.996118 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:23:17.996082 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38a6ae92_90b3_49c8_8300_84a8cb1cd68e.slice/crio-519a17a2d37dbb2d7ebf234f32ad6ec10970951f64a531abdf0528ee5a804f24 WatchSource:0}: Error finding container 519a17a2d37dbb2d7ebf234f32ad6ec10970951f64a531abdf0528ee5a804f24: Status 404 returned error can't find the container with id 519a17a2d37dbb2d7ebf234f32ad6ec10970951f64a531abdf0528ee5a804f24 Jul 07 00:23:17.997897 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:17.997878 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:23:18.830979 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:18.830870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmw4w/must-gather-n5x62" event={"ID":"38a6ae92-90b3-49c8-8300-84a8cb1cd68e","Type":"ContainerStarted","Data":"5b364723b73d61095686bfb2da1a666bf8e2886b9bfd70c53cbc9265a5435f16"} Jul 07 00:23:18.830979 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:18.830922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmw4w/must-gather-n5x62" event={"ID":"38a6ae92-90b3-49c8-8300-84a8cb1cd68e","Type":"ContainerStarted","Data":"519a17a2d37dbb2d7ebf234f32ad6ec10970951f64a531abdf0528ee5a804f24"} Jul 07 00:23:19.840302 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:19.840245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmw4w/must-gather-n5x62" event={"ID":"38a6ae92-90b3-49c8-8300-84a8cb1cd68e","Type":"ContainerStarted","Data":"79b3bb2332153b6c2a67f60f884b7d1b1a81dc3e001773a5128aef65d17487f1"} Jul 07 00:23:19.859142 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:19.859090 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tmw4w/must-gather-n5x62" podStartSLOduration=2.181603933 podStartE2EDuration="2.85907411s" totalImagesPullingTime="677.470177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:23:19.857686625 +0000 UTC m=+2429.884591855" watchObservedRunningTime="2026-07-07 00:23:19.85907411 +0000 UTC m=+2429.885979338" Jul 07 00:23:20.315198 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:20.315159 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7rs8n_fc3c82fa-4033-4a51-aeb3-3fb8441ac7cd/global-pull-secret-syncer/0.log" Jul 07 00:23:20.415366 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:20.415339 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" Jul 07 00:23:20.517088 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:20.517058 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-183.ec2.internal_9064f225fd7e0619db91cfef5c8bbc9d/haproxy/0.log" Jul 07 00:23:24.906394 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:24.906367 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-r5t95_d1d94964-164e-449e-b637-2a683b6d6620/manager/0.log" Jul 07 00:23:25.010642 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:25.010595 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hdjb4_aa521a3e-2db8-4adb-bb07-ecba685763f3/registry-server/0.log" Jul 07 00:23:27.057877 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:27.057837 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-84wcc_870abde4-501e-4a82-837d-abbb43a831cd/cluster-monitoring-operator/0.log" Jul 07 00:23:27.336417 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:27.336333 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qx4zk_a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0/node-exporter/0.log" Jul 07 00:23:27.370985 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:27.370960 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qx4zk_a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0/kube-rbac-proxy/0.log" Jul 07 00:23:27.400213 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:27.400175 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qx4zk_a78d8a51-0d3d-4b8b-a93e-e2b41787f4c0/init-textfile/0.log" Jul 07 00:23:29.568550 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:29.568522 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-6rmbv_7b775ca7-c969-4e1c-a836-bcdaa9803f68/networking-console-plugin/0.log" Jul 07 00:23:30.160064 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.160026 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/1.log" Jul 07 00:23:30.170475 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.170442 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sf98j_1e5f900f-d24d-4c1f-a353-b1f26165a5eb/console-operator/2.log" Jul 07 00:23:30.459955 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.459873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:30.473960 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.473931 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf"] Jul 07 00:23:30.481744 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.481716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.486341 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.486314 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf"] Jul 07 00:23:30.535808 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.535772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-lib-modules\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.536068 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.536036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-sys\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.536174 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.536078 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22sl\" (UniqueName: \"kubernetes.io/projected/7dfc6a09-9606-4ef6-97ba-c149d032ed82-kube-api-access-h22sl\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.536174 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.536140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-podres\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.536174 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.536169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-proc\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.637599 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.637563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-sys\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.637614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22sl\" (UniqueName: \"kubernetes.io/projected/7dfc6a09-9606-4ef6-97ba-c149d032ed82-kube-api-access-h22sl\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.637726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-podres\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.637741 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-sys\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.637759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-proc\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.637843 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-proc\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.637963 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-podres\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638065 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.638046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-lib-modules\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.638327 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.638136 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dfc6a09-9606-4ef6-97ba-c149d032ed82-lib-modules\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.645796 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.645754 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22sl\" (UniqueName: \"kubernetes.io/projected/7dfc6a09-9606-4ef6-97ba-c149d032ed82-kube-api-access-h22sl\") pod \"perf-node-gather-daemonset-lrbtf\" (UID: \"7dfc6a09-9606-4ef6-97ba-c149d032ed82\") " pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.795169 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.795141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:30.801388 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.801358 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-bjj78_bbec8d32-f9bf-44fe-b4ec-b0b35dfff7d1/download-server/0.log" Jul 07 00:23:31.001573 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:30.999179 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf"] Jul 07 00:23:31.003396 ip-10-0-132-183 kubenswrapper[2564]: W0707 00:23:31.003352 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7dfc6a09_9606_4ef6_97ba_c149d032ed82.slice/crio-2a7a332c07886361fdbc29095356162f284386ddf950ef6e6c2747eb76cdd9c0 WatchSource:0}: Error finding container 2a7a332c07886361fdbc29095356162f284386ddf950ef6e6c2747eb76cdd9c0: Status 404 returned error can't find the container with id 2a7a332c07886361fdbc29095356162f284386ddf950ef6e6c2747eb76cdd9c0 Jul 07 00:23:31.476437 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:31.476364 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-ghzlf_89b36ac3-0f96-40df-9c23-718e2d608e87/volume-data-source-validator/0.log" Jul 07 00:23:31.934926 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:31.934877 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" event={"ID":"7dfc6a09-9606-4ef6-97ba-c149d032ed82","Type":"ContainerStarted","Data":"7c64b64c8adb3e12f5d52c6fbe2de9bcc563cce8460ea39f4e047c6b298921bb"} Jul 07 00:23:31.935415 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:31.934954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" event={"ID":"7dfc6a09-9606-4ef6-97ba-c149d032ed82","Type":"ContainerStarted","Data":"2a7a332c07886361fdbc29095356162f284386ddf950ef6e6c2747eb76cdd9c0"} Jul 07 00:23:31.935810 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:31.935788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:31.955171 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:31.955113 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" podStartSLOduration=1.955093834 podStartE2EDuration="1.955093834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 00:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:23:31.952023405 +0000 UTC m=+2441.978928637" watchObservedRunningTime="2026-07-07 00:23:31.955093834 +0000 UTC m=+2441.981999065" Jul 07 00:23:32.611669 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:32.611619 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wpbdz_7d410004-c4aa-4f89-9376-6ca83adf6ace/dns/0.log" Jul 07 00:23:32.633604 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:32.633574 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wpbdz_7d410004-c4aa-4f89-9376-6ca83adf6ace/kube-rbac-proxy/0.log" Jul 07 00:23:32.657122 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:32.657081 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-brd86_9bc12b8c-a240-473d-a57c-b9cd825797ef/dns-node-resolver/0.log" Jul 07 00:23:33.228479 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:33.228444 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29723040-7vx28_9329f926-1025-4c26-bbb9-bbe6ef08a871/image-pruner/0.log" Jul 07 00:23:33.313769 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:33.313740 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sd5wt_dbafaea7-f7b6-4dd8-b32a-fd38510ed29d/node-ca/0.log" Jul 07 00:23:34.665138 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:34.665109 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lw5ck_92a1830b-dcd2-4e3d-b888-ef32b1f84442/payload-processing/0.log" Jul 07 00:23:34.688795 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:34.688224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d86c67597-d9q6b_0522ae39-76b7-493c-9a01-2913bf880b08/router/0.log" Jul 07 00:23:35.348229 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:35.348187 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b49kq_d5287318-0b55-4737-ac2f-be4aa83b35ba/serve-healthcheck-canary/0.log" Jul 07 00:23:35.853060 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:35.853011 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jzx9r_438b86b7-b5b2-4243-a133-af4c044f9b57/insights-operator/0.log" Jul 07 00:23:35.856028 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:35.855999 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jzx9r_438b86b7-b5b2-4243-a133-af4c044f9b57/insights-operator/1.log" Jul 07 00:23:35.878135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:35.878103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7hxbm_7dc94e3c-1c5b-4495-80d4-ff4288bdedd2/kube-rbac-proxy/0.log" Jul 07 00:23:35.897092 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:35.897064 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7hxbm_7dc94e3c-1c5b-4495-80d4-ff4288bdedd2/exporter/0.log" Jul 07 00:23:35.918473 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:35.918447 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7hxbm_7dc94e3c-1c5b-4495-80d4-ff4288bdedd2/extractor/0.log" Jul 07 00:23:38.353477 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.353441 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5ddd4945c8-52wgr_4cf12e6e-add2-4904-97a1-32e9f5db5af2/maas-api/0.log" Jul 07 00:23:38.372309 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.372276 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723040-94xpf_c971ef3f-1281-4d73-a14f-cdd896dff546/cleanup/0.log" Jul 07 00:23:38.564950 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.564916 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-696cfbbf9b-pm4gt_eaddac64-3801-478c-9737-f59ba2f8c621/manager/0.log" Jul 07 00:23:38.690371 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.690254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:38.756380 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.756336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qrzv_478986e2-196e-4b09-8536-4f2d903d6c39/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:38.778819 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.778794 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6688846b76-hwrjr_a1510dbb-cae6-4353-8bad-e7ddda73a8ac/manager/0.log" Jul 07 00:23:38.795705 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.795677 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6688846b76-hwrjr_a1510dbb-cae6-4353-8bad-e7ddda73a8ac/copy-manifests/0.log" Jul 07 00:23:38.910246 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.910216 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-rwzgz_c56466fa-f92c-42f2-9756-494d2003a9d6/postgres/0.log" Jul 07 00:23:38.960011 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:38.959931 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tmw4w/perf-node-gather-daemonset-lrbtf" Jul 07 00:23:40.254135 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:40.254105 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6557895b7f-2czcg_3a39751b-8342-4db8-af20-99546112cc84/manager/0.log" Jul 07 00:23:41.360835 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:41.360799 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lw5ck_92a1830b-dcd2-4e3d-b888-ef32b1f84442/payload-processing/0.log" Jul 07 00:23:41.378755 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:41.378721 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d86c67597-d9q6b_0522ae39-76b7-493c-9a01-2913bf880b08/router/0.log" Jul 07 00:23:42.879044 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:42.879013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lw5ck_92a1830b-dcd2-4e3d-b888-ef32b1f84442/payload-processing/0.log" Jul 07 00:23:42.898803 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:42.898777 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d86c67597-d9q6b_0522ae39-76b7-493c-9a01-2913bf880b08/router/0.log" Jul 07 00:23:45.577499 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:45.577465 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7xqln_050f8482-af52-4f04-aa9c-328646073ab0/migrator/0.log" Jul 07 00:23:45.600296 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:45.600272 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7xqln_050f8482-af52-4f04-aa9c-328646073ab0/graceful-termination/0.log" Jul 07 00:23:45.959675 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:45.959573 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-b97bk_539120a9-c4eb-424e-942b-f7f2047712b7/kube-storage-version-migrator-operator/1.log" Jul 07 00:23:45.961330 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:45.961307 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-b97bk_539120a9-c4eb-424e-942b-f7f2047712b7/kube-storage-version-migrator-operator/0.log" Jul 07 00:23:46.993970 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:46.993936 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-24bw4_52506102-c9ef-4fd1-b154-19c1a64604f3/kube-multus/0.log" Jul 07 00:23:47.018931 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.018905 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rf6b_82b7c2c3-750c-4b5f-bc86-dd23f6292ad5/kube-multus-additional-cni-plugins/0.log" Jul 07 00:23:47.043467 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.043430 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rf6b_82b7c2c3-750c-4b5f-bc86-dd23f6292ad5/egress-router-binary-copy/0.log" Jul 07 00:23:47.063539 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.063506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rf6b_82b7c2c3-750c-4b5f-bc86-dd23f6292ad5/cni-plugins/0.log" Jul 07 00:23:47.084351 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.084326 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rf6b_82b7c2c3-750c-4b5f-bc86-dd23f6292ad5/bond-cni-plugin/0.log" Jul 07 00:23:47.105294 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.105268 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rf6b_82b7c2c3-750c-4b5f-bc86-dd23f6292ad5/routeoverride-cni/0.log" Jul 07 00:23:47.125084 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.125039 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rf6b_82b7c2c3-750c-4b5f-bc86-dd23f6292ad5/whereabouts-cni-bincopy/0.log" Jul 07 00:23:47.144593 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.144568 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rf6b_82b7c2c3-750c-4b5f-bc86-dd23f6292ad5/whereabouts-cni/0.log" Jul 07 00:23:47.582494 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.582465 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6lqdv_670307dd-877f-4f25-ad8c-3b6bb6ce1134/network-metrics-daemon/0.log" Jul 07 00:23:47.597798 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:47.597768 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6lqdv_670307dd-877f-4f25-ad8c-3b6bb6ce1134/kube-rbac-proxy/0.log" Jul 07 00:23:48.642747 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.642718 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-controller/0.log" Jul 07 00:23:48.662207 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.662182 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/0.log" Jul 07 00:23:48.662207 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.662219 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovn-acl-logging/1.log" Jul 07 00:23:48.684276 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.684252 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/kube-rbac-proxy-node/0.log" Jul 07 00:23:48.705608 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.705577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/kube-rbac-proxy-ovn-metrics/0.log" Jul 07 00:23:48.724942 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.724906 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/northd/0.log" Jul 07 00:23:48.747435 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.747410 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/nbdb/0.log" Jul 07 00:23:48.768098 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.768075 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/sbdb/0.log" Jul 07 00:23:48.857293 ip-10-0-132-183 kubenswrapper[2564]: I0707 00:23:48.857214 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9pgh_ceff6ec7-0ad5-413a-b36b-124caf4231af/ovnkube-controller/0.log"