Jul 09 00:55:34.555835 ip-10-0-134-128 systemd[1]: Starting Kubernetes Kubelet... Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:34.995063 2543 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001568 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001580 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001583 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001587 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001590 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001593 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001596 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 00:55:35.034573 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001599 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001602 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001604 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001607 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001610 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001613 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001622 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001625 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001628 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001631 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001634 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001636 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001639 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001642 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001645 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001648 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001651 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001654 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001657 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001660 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 00:55:35.035398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001662 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001665 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001668 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001671 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001674 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001676 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001679 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001682 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001684 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001687 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001689 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001692 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001695 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001697 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001700 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001703 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001706 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001709 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001712 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001715 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 00:55:35.036268 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001717 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001720 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001723 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001727 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001731 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001733 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001736 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001739 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001741 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001745 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001747 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001751 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001755 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001759 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001761 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001764 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001766 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001769 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001771 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 00:55:35.037058 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001774 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001776 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001778 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001781 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001784 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001787 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001790 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001793 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001795 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001798 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001802 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001805 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001808 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001811 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001814 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001817 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001820 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001822 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001825 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 00:55:35.037867 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001827 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001830 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001832 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001835 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001837 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001839 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001842 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001844 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001847 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001849 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001852 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001854 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001857 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.001859 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002572 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002581 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002585 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002588 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002590 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 00:55:35.038672 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002595 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002599 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002603 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002606 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002608 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002611 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002613 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002616 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002619 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002621 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002624 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002626 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002629 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002631 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002634 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002637 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002641 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002643 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002646 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002649 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 00:55:35.039454 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002651 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002655 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002658 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002661 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002664 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002666 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002669 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002671 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002674 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002676 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002679 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002681 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002684 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002686 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002689 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002691 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002695 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002697 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002700 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 00:55:35.040229 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002702 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002706 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002709 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002711 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002714 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002717 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002719 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002722 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002724 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002727 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002730 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002733 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002735 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002738 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002741 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002743 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002746 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002748 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002751 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002753 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 00:55:35.043847 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002756 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002758 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002760 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002763 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002765 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002767 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002770 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002773 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002776 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002778 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002781 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002784 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002786 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002789 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002792 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002794 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002799 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002802 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002804 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002807 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 00:55:35.045048 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002809 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002812 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002815 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002817 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002820 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002822 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002825 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002827 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002830 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002832 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002834 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002837 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002840 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002842 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.002844 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003201 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003205 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003208 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003211 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003218 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 00:55:35.045849 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003220 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003223 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003226 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003228 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003231 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003233 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003236 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003238 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003243 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003246 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003248 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003251 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003253 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003256 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003259 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003262 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003264 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003267 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003270 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 00:55:35.047464 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003273 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003275 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003278 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003280 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003283 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003285 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003288 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003291 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003293 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003295 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003298 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003300 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003303 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003305 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003308 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003310 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003313 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003315 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003317 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003320 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 00:55:35.048261 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003323 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003329 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003331 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003334 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003336 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003339 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003343 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003346 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003348 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003350 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003353 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003355 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003358 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003361 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003363 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003366 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003368 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003371 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003375 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 00:55:35.049065 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003378 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003381 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003384 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003386 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003389 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003391 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003394 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003397 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003399 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003401 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003404 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003408 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003411 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003414 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003418 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003421 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003423 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003426 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003428 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003432 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 00:55:35.050309 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003434 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003437 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003439 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003442 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003444 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003447 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003449 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003452 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003454 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003457 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003459 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003461 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003464 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003466 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003468 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.003471 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003555 2543 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003566 2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003573 2543 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003577 2543 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003582 2543 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 00:55:35.051118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003585 2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003589 2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003593 2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003596 2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003599 2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003604 2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003607 2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003610 2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003613 2543 flags.go:64] FLAG: --cgroup-root="" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003616 2543 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003620 2543 flags.go:64] FLAG: --client-ca-file="" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003622 2543 flags.go:64] FLAG: --cloud-provider="external" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003625 2543 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003629 2543 flags.go:64] FLAG: --cluster-domain="" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003632 2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003634 2543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003637 2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003641 2543 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003644 2543 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003647 2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003650 2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003653 2543 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003655 2543 flags.go:64] FLAG: --contention-profiling="false" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003659 2543 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 00:55:35.052692 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003661 2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003664 2543 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003667 2543 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003671 2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003674 2543 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003677 2543 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003679 2543 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003682 2543 flags.go:64] FLAG: --enable-server="true" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003685 2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003689 2543 flags.go:64] FLAG: --event-burst="100" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003692 2543 flags.go:64] FLAG: --event-qps="50" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003695 2543 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003698 2543 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003703 2543 flags.go:64] FLAG: --eviction-hard="" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003706 2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003709 2543 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003712 2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003715 2543 flags.go:64] FLAG: --eviction-soft="" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003720 2543 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003722 2543 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003726 2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003728 2543 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003732 2543 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003735 2543 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003737 2543 flags.go:64] FLAG: --feature-gates="" Jul 09 00:55:35.053501 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003741 2543 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003744 2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003746 2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003749 2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003752 2543 flags.go:64] FLAG: --healthz-port="10248" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003755 2543 flags.go:64] FLAG: --help="false" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003758 2543 flags.go:64] FLAG: --hostname-override="ip-10-0-134-128.ec2.internal" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003761 2543 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003764 2543 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003767 2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003770 2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003773 2543 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003776 2543 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003779 2543 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003781 2543 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003784 2543 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003787 2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003790 2543 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003793 2543 flags.go:64] FLAG: --kube-reserved="" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003796 2543 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003800 2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003803 2543 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003806 2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003808 2543 flags.go:64] FLAG: --lock-file="" Jul 09 00:55:35.054948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003811 2543 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003815 2543 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003818 2543 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003823 2543 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003826 2543 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003828 2543 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003831 2543 flags.go:64] FLAG: --logging-format="text" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003834 2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003837 2543 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003840 2543 flags.go:64] FLAG: --manifest-url="" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003842 2543 flags.go:64] FLAG: --manifest-url-header="" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003846 2543 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003849 2543 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003853 2543 flags.go:64] FLAG: --max-pods="110" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003856 2543 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003859 2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003862 2543 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003864 2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003867 2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003870 2543 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003873 2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003879 2543 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003882 2543 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003885 2543 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 00:55:35.055726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003888 2543 flags.go:64] FLAG: --pod-cidr="" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003891 2543 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003896 2543 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003899 2543 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003903 2543 flags.go:64] FLAG: --pods-per-core="0" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003906 2543 flags.go:64] FLAG: --port="10250" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003909 2543 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003911 2543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bcef657cf5faa861" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003914 2543 flags.go:64] FLAG: --qos-reserved="" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003917 2543 flags.go:64] FLAG: --read-only-port="10255" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003921 2543 flags.go:64] FLAG: --register-node="true" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003924 2543 flags.go:64] FLAG: --register-with-taints="" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003928 2543 flags.go:64] FLAG: --registry-burst="10" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003930 2543 flags.go:64] FLAG: --registry-qps="5" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003933 2543 flags.go:64] FLAG: --reserved-cpus="" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003936 2543 flags.go:64] FLAG: --reserved-memory="" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003939 2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003942 2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003945 2543 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003948 2543 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003951 2543 flags.go:64] FLAG: --runonce="false" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003953 2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003956 2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003959 2543 flags.go:64] FLAG: --seccomp-default="false" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003962 2543 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 00:55:35.057369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003965 2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003967 2543 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003970 2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003973 2543 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003976 2543 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003981 2543 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003983 2543 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003986 2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003989 2543 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003992 2543 flags.go:64] FLAG: --system-cgroups="" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.003995 2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004001 2543 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004004 2543 flags.go:64] FLAG: --tls-cert-file="" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004007 2543 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004010 2543 flags.go:64] FLAG: --tls-min-version="" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004013 2543 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004016 2543 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004020 2543 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004023 2543 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004026 2543 flags.go:64] FLAG: --v="2" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004029 2543 flags.go:64] FLAG: --version="false" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004033 2543 flags.go:64] FLAG: --vmodule="" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004036 2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.004039 2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004202 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 00:55:35.058604 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004216 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004220 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004233 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004237 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004240 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004243 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004247 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004250 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004253 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004255 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004258 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004261 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004264 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004267 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004269 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004272 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004275 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004277 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004280 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004283 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 00:55:35.059850 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004286 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004290 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004292 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004295 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004298 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004302 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004305 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004309 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004314 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004317 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004321 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004324 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004328 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004330 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004333 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004336 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004338 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004341 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004345 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 00:55:35.061071 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004348 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004351 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004353 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004356 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004358 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004362 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004366 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004369 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004371 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004374 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004377 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004379 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004382 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004384 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004388 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004390 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004393 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004395 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004398 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 00:55:35.063373 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004400 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004403 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004405 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004408 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004410 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004414 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004416 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004419 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004422 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004425 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004427 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004430 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004435 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004437 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004440 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004443 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004445 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004447 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004450 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004452 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 00:55:35.064804 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004455 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004457 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004460 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004462 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004465 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004468 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004470 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004472 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004475 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004477 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004480 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004482 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004498 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004501 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004504 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004506 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004510 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004513 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004516 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 00:55:35.067237 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.004518 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 00:55:35.066048 ip-10-0-134-128 systemd[1]: Started Kubernetes Kubelet. Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.005163 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.011326 2543 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.011339 2543 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.012132 2543 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.012139 2543 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012194 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012199 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012202 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012205 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012208 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012211 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012214 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012217 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012220 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 00:55:35.068398 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012223 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012225 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012228 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012231 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012234 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012236 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012239 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012241 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012244 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012247 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012249 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012252 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012256 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012260 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012263 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012267 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012270 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012273 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012276 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 00:55:35.072479 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012279 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012283 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012286 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012289 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012292 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012295 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012297 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012300 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012303 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012305 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012308 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012310 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012313 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012315 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012317 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012320 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012322 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012325 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012328 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012330 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 00:55:35.073238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012333 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012335 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012338 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012340 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012343 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012346 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012348 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012351 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012353 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012356 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012358 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012361 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012364 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012367 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012369 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012371 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012374 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012377 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012379 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012381 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 00:55:35.169959 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012384 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012386 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012389 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012391 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012394 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012396 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012399 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012403 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012406 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012409 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012412 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012415 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012417 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012420 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012423 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012425 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012428 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012431 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012433 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 00:55:35.170651 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012436 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012438 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012441 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012444 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012446 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012449 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012452 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012454 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012457 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012459 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012461 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012464 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.012469 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012604 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012611 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 00:55:35.171415 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012615 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012618 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012621 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012624 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012627 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012630 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012632 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012635 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012637 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012640 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012642 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012645 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012647 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012650 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012652 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012655 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012658 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012660 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012663 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012665 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 00:55:35.172188 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012668 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012670 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012673 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012676 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012681 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012683 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012686 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012689 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012691 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012693 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012696 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012699 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012701 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012705 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012708 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012711 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012714 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012717 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012719 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 00:55:35.174948 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012722 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012725 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012728 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012732 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012734 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012737 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012740 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012742 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012745 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012747 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012750 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012752 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012755 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012757 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012759 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012762 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012764 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012768 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012771 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012774 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 00:55:35.176126 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012776 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012779 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012782 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012784 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012787 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012789 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012792 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012794 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012797 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012801 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012805 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012809 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012813 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012815 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012818 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012822 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012824 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012827 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012829 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 00:55:35.177709 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012832 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012834 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012837 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012839 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012842 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012844 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012846 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012849 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012851 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012854 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012858 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012860 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012864 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012866 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012869 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012871 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012886 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012889 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 00:55:35.178481 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:35.012892 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.012897 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.012987 2543 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.014874 2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.016226 2543 server.go:1019] "Starting client certificate rotation" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.016319 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.016347 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.018686 2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.023381 2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.030623 2543 log.go:25] "Validated CRI v1 runtime API" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.034934 2543 log.go:25] "Validated CRI v1 image API" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.035755 2543 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 00:55:35.190211 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.038047 2543 fs.go:135] Filesystem UUIDs: map[10f49e88-4c48-49c5-8ee9-c1cd7505c29c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d64f81a1-2ffb-4b25-9a16-35a698b47e5b:/dev/nvme0n1p4] Jul 09 00:55:35.190985 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.038059 2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 00:55:35.192572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.043371 2543 manager.go:216] Machine: {Timestamp:2026-07-09 00:55:35.041767175 +0000 UTC m=+0.359760954 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109211 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff9091f90d887247f8f193f278320 SystemUUID:ec2ff909-1f90-d887-247f-8f193f278320 BootID:763118d9-950c-4096-b9ae-70d86fa7e655 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e7:90:02:32:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e7:90:02:32:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:d9:f1:81:d7:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 00:55:35.192572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.043468 2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 00:55:35.192572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.043550 2543 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 00:55:35.192572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.046546 2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 00:55:35.192572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.047003 2543 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 00:55:35.192572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.047034 2543 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 00:55:35.192572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.047715 2543 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.047729 2543 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.047740 2543 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.048317 2543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.049475 2543 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.053597 2543 state_mem.go:36] "Initialized new in-memory state store" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.053730 2543 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.053763 2543 kubelet.go:486] "Attempting to sync node with API server" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.053775 2543 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.053788 2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.053800 2543 kubelet.go:398] "Adding apiserver pod source" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.053810 2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.054152 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.054173 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.056424 2543 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.058404 2543 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.058440 2543 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059472 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059502 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059510 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059517 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059529 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059536 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059546 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059553 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059563 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059570 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059589 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.059597 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.060413 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.060427 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.063710 2543 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:35.063818 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:35.063938 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.065317 2543 server.go:1268] "Started kubelet" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.065385 2543 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.065509 2543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.065567 2543 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.067118 2543 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.067413 2543 server.go:310] "Adding debug handlers to kubelet server" Jul 09 00:55:35.192948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.069133 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s7wqg" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:35.069659 2543 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.073809 2543 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.073822 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:35.073895 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074415 2543 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074431 2543 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074417 2543 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074590 2543 reconstruct.go:97] "Volume reconstruction finished" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074596 2543 reconciler.go:29] "Reconciler: start to sync state" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074689 2543 factory.go:55] Registering systemd factory Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074701 2543 factory.go:223] Registration of the systemd container factory successfully Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074963 2543 factory.go:153] Registering CRI-O factory Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.074973 2543 factory.go:223] Registration of the crio container factory successfully Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.075010 2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.075028 2543 factory.go:103] Registering Raw factory Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.075042 2543 manager.go:1196] Started watching for new ooms in manager Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.075434 2543 manager.go:349] Starting recovery of all containers Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:35.075936 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s7wqg" Jul 09 00:55:35.195304 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:35.173960 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.339204 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.314405 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.457928 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.414689 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.574138 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.514943 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.680361 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.615542 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.715891 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.715846 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.763636 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.763385 2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 00:55:36.764099 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.764080 2543 manager.go:354] Recovery completed Jul 09 00:55:36.766360 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.766314 2543 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 09 00:55:36.766530 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.766510 2543 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-128.ec2.internal\" not found" node="ip-10-0-134-128.ec2.internal" Jul 09 00:55:36.770354 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.770341 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 00:55:36.772335 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.772319 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 00:55:36.772401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.772348 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 00:55:36.772401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.772362 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 00:55:36.772903 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.772889 2543 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 00:55:36.772903 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.772896 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 00:55:36.772903 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.772902 2543 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 00:55:36.773026 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.772921 2543 state_mem.go:36] "Initialized new in-memory state store" Jul 09 00:55:36.773519 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.773507 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 00:55:36.773577 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.773530 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 00:55:36.773577 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.773539 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 00:55:36.774897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.774886 2543 policy_none.go:49] "None policy: Start" Jul 09 00:55:36.774942 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.774900 2543 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 00:55:36.774942 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.774910 2543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 00:55:36.776802 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.776788 2543 policy_none.go:47] "Start" Jul 09 00:55:36.811201 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.811185 2543 manager.go:337] "Starting Device Plugin manager" Jul 09 00:55:36.811275 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.811211 2543 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 00:55:36.811275 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.811220 2543 server.go:87] "Starting device plugin registration server" Jul 09 00:55:36.811413 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.811401 2543 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 00:55:36.811463 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.811415 2543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 00:55:36.811514 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.811502 2543 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 00:55:36.811592 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.811574 2543 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 00:55:36.811592 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.811587 2543 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 00:55:36.812082 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.812064 2543 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 00:55:36.812144 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.812117 2543 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.911860 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.911841 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 00:55:36.913952 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.912722 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 00:55:36.913952 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.912753 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 00:55:36.913952 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.912766 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 00:55:36.913952 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.912788 2543 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-128.ec2.internal" Jul 09 00:55:36.921050 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.921031 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 00:55:36.921262 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.921250 2543 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-128.ec2.internal" Jul 09 00:55:36.921301 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.921271 2543 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-128.ec2.internal\": node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:36.922199 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.922183 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 00:55:36.922251 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.922228 2543 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 00:55:36.922282 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.922252 2543 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 00:55:36.922312 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.922288 2543 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 00:55:36.924032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:36.924014 2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 00:55:36.968723 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:36.968679 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Jul 09 00:55:37.023157 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.023125 2543 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal"] Jul 09 00:55:37.023239 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.023204 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 00:55:37.023983 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.023956 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 00:55:37.023983 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.023981 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 00:55:37.024134 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.023990 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 00:55:37.026197 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.026185 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 00:55:37.026315 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.026300 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.026381 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.026333 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 00:55:37.026837 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.026822 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 00:55:37.026919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.026853 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 00:55:37.026919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.026864 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 00:55:37.028186 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.028171 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 00:55:37.028256 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.028196 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 00:55:37.028256 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.028204 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 00:55:37.030263 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.030243 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.030314 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.030266 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 00:55:37.031512 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.031490 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 00:55:37.031602 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.031523 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 00:55:37.031602 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.031538 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 00:55:37.034417 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.034401 2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 00:55:37.042164 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.042150 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.047421 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.047408 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.055284 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.055272 2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 00:55:37.055410 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.055386 2543 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.134.128:52516->100.30.55.184:6443: use of closed network connection" Jul 09 00:55:37.055521 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.055407 2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 00:55:37.055521 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.055415 2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 00:55:37.055521 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.055433 2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 00:55:37.055521 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.055357 2543 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a0533aacda3074cd0b37a929f0eb7e36-c01e105f30be9586.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-128.ec2.internal.18c0790a1c6e31e8\": read tcp 10.0.134.128:52516->100.30.55.184:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-128.ec2.internal.18c0790a1c6e31e8 default 6948 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-128.ec2.internal,UID:ip-10-0-134-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-128.ec2.internal,},FirstTimestamp:2026-07-09 00:55:36 +0000 UTC,LastTimestamp:2026-07-09 00:55:37.026838059 +0000 UTC m=+1.304676708,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-128.ec2.internal,}" Jul 09 00:55:37.055521 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.055437 2543 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a0533aacda3074cd0b37a929f0eb7e36-c01e105f30be9586.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.134.128:52516->100.30.55.184:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.055720 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.055436 2543 kubelet.go:3250] "Failed creating a mirror pod" err="unexpected error when reading response body. Please retry. Original error: read tcp 10.0.134.128:52516->100.30.55.184:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.094237 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.094220 2543 apiserver.go:52] "Watching apiserver" Jul 09 00:55:37.110034 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.110014 2543 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 00:55:37.110268 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.110253 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-klfjr","openshift-image-registry/node-ca-s55gv","openshift-multus/multus-9qmvg","openshift-multus/multus-additional-cni-plugins-wjb6p","openshift-network-operator/iptables-alerter-dtxxn","kube-system/konnectivity-agent-649tx","openshift-multus/network-metrics-daemon-69lnl","openshift-network-diagnostics/network-check-target-2mpcm","kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal"] Jul 09 00:55:37.112518 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.112505 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.114368 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.114348 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.114368 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.114364 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 00:55:37.115078 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.114810 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.115078 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.114838 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 00:55:37.115078 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.114871 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 00:55:37.115078 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.114939 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wksbq\"" Jul 09 00:55:37.116815 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.116797 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 00:55:37.116990 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.116972 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.117307 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.117242 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 00:55:37.117764 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.117745 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v9fsd\"" Jul 09 00:55:37.117862 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.117829 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 00:55:37.119045 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119025 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 00:55:37.119295 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119258 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 00:50:36 +0000 UTC" deadline="2027-12-04 14:59:43.083857862 +0000 UTC" Jul 09 00:55:37.119354 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119296 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12326h4m5.964564874s" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119441 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-netns\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119481 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-multus-certs\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119517 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtv7\" (UniqueName: \"kubernetes.io/projected/8afaa9e7-52cc-4050-9a84-03208d9d39c2-kube-api-access-chtv7\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119573 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fd838a4b982cf3dfd5cfb41e87f7285-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"3fd838a4b982cf3dfd5cfb41e87f7285\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119596 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd838a4b982cf3dfd5cfb41e87f7285-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"3fd838a4b982cf3dfd5cfb41e87f7285\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119624 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53d14f6-bfe9-4d19-ad46-d16aea78249c-tmp-dir\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119742 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6tp\" (UniqueName: \"kubernetes.io/projected/b53d14f6-bfe9-4d19-ad46-d16aea78249c-kube-api-access-2t6tp\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119777 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-cni-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119813 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119822 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8afaa9e7-52cc-4050-9a84-03208d9d39c2-cni-binary-copy\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119858 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-socket-dir-parent\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119943 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-cni-multus\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.119985 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-kubelet\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120015 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-hostroot\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120060 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120065 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-host\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120070 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120089 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-system-cni-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120154 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-os-release\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120186 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hk7z7\"" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120187 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-k8s-cni-cncf-io\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120226 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-daemon-config\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120264 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c46fe0b45721f50ed4b3952298077b2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-128.ec2.internal\" (UID: \"1c46fe0b45721f50ed4b3952298077b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120315 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b53d14f6-bfe9-4d19-ad46-d16aea78249c-hosts-file\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120356 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgg8\" (UniqueName: \"kubernetes.io/projected/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-kube-api-access-5zgg8\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120389 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-conf-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.120897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120437 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-cnibin\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.121366 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.121052 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.121366 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.120527 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-cni-bin\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.121631 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.121608 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-etc-kubernetes\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.121746 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.121665 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-serviceca\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.122167 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.122152 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:37.122246 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.122224 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:37.123761 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.123742 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s29xf\"" Jul 09 00:55:37.123761 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.123752 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 00:55:37.123890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.123768 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 00:55:37.124285 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.124268 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:37.124358 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.124338 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:37.125204 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.125188 2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 00:55:37.126447 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.126435 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.128309 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.128295 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t6w65\"" Jul 09 00:55:37.128396 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.128380 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 00:55:37.128508 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.128492 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.128551 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.128526 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 00:55:37.128653 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.128626 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 00:55:37.130781 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.130763 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 00:55:37.130867 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.130768 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ccsk5\"" Jul 09 00:55:37.130929 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.130875 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 00:55:37.135819 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.135795 2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 00:55:37.135915 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.135848 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.135985 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.135929 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal"] Jul 09 00:55:37.141736 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.141720 2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 00:55:37.148612 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.148597 2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 00:55:37.148688 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.148636 2543 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-128.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.149260 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.149244 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-ghvhc"] Jul 09 00:55:37.151625 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.151610 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz"] Jul 09 00:55:37.151732 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.151718 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.152628 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.152614 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jx675" Jul 09 00:55:37.154114 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.154103 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.154508 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.154484 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 00:55:37.154579 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.154541 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 00:55:37.154632 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.154618 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v6tkk\"" Jul 09 00:55:37.156081 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.156067 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 00:55:37.156284 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.156270 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 00:55:37.156366 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.156354 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 00:55:37.156522 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.156510 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-28wfm\"" Jul 09 00:55:37.162788 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.162773 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jx675" Jul 09 00:55:37.168777 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.168761 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-thjzt"] Jul 09 00:55:37.171583 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.171285 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.171583 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.171350 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:37.215347 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.215333 2543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 00:55:37.222352 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222306 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-os-release\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222352 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222336 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c46fe0b45721f50ed4b3952298077b2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-128.ec2.internal\" (UID: \"1c46fe0b45721f50ed4b3952298077b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222363 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c95d9938-3b58-44fd-a335-b743e9772e5e-konnectivity-ca\") pod \"konnectivity-agent-649tx\" (UID: \"c95d9938-3b58-44fd-a335-b743e9772e5e\") " pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222386 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222410 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-registration-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222417 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-os-release\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222433 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9e01ea33-3313-47a6-a2b3-93a28babfdbd-iptables-alerter-script\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222417 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c46fe0b45721f50ed4b3952298077b2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-128.ec2.internal\" (UID: \"1c46fe0b45721f50ed4b3952298077b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222458 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-cni-bin\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222483 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-serviceca\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222515 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-cni-bin\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222519 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-var-lib-kubelet\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222561 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94802df-4092-42e6-ac8c-c0408a4b415a-tmp\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222578 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvhp\" (UniqueName: \"kubernetes.io/projected/f94802df-4092-42e6-ac8c-c0408a4b415a-kube-api-access-lzvhp\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222596 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fd838a4b982cf3dfd5cfb41e87f7285-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"3fd838a4b982cf3dfd5cfb41e87f7285\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222613 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222636 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-socket-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222676 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-system-cni-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222698 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fd838a4b982cf3dfd5cfb41e87f7285-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"3fd838a4b982cf3dfd5cfb41e87f7285\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222701 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgg8\" (UniqueName: \"kubernetes.io/projected/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-kube-api-access-5zgg8\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222741 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-etc-kubernetes\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222748 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-system-cni-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222782 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-etc-kubernetes\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.222814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222789 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysctl-d\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222816 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6tp\" (UniqueName: \"kubernetes.io/projected/b53d14f6-bfe9-4d19-ad46-d16aea78249c-kube-api-access-2t6tp\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222837 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cnibin\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222948 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb98w\" (UniqueName: \"kubernetes.io/projected/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-kube-api-access-gb98w\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222977 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-cni-multus\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.222993 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c95d9938-3b58-44fd-a335-b743e9772e5e-agent-certs\") pod \"konnectivity-agent-649tx\" (UID: \"c95d9938-3b58-44fd-a335-b743e9772e5e\") " pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223011 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-os-release\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223036 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-k8s-cni-cncf-io\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223047 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-cni-multus\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223073 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223096 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-k8s-cni-cncf-io\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223106 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-device-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223131 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223153 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-conf-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223168 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-run\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223182 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-tuned\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223201 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b581286c-01e6-45a0-a760-5bb689a8b8e3-dbus\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223219 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-conf-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223234 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-netns\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223252 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd838a4b982cf3dfd5cfb41e87f7285-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"3fd838a4b982cf3dfd5cfb41e87f7285\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223268 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53d14f6-bfe9-4d19-ad46-d16aea78249c-tmp-dir\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223253 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-netns\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223289 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-sys\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223291 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd838a4b982cf3dfd5cfb41e87f7285-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"3fd838a4b982cf3dfd5cfb41e87f7285\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223312 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-lib-modules\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223356 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8afaa9e7-52cc-4050-9a84-03208d9d39c2-cni-binary-copy\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223380 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-host\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223408 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-sys-fs\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.223637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223426 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-system-cni-dir\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223431 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-host\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223445 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnj2\" (UniqueName: \"kubernetes.io/projected/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-kube-api-access-tdnj2\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223464 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-daemon-config\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223479 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b53d14f6-bfe9-4d19-ad46-d16aea78249c-hosts-file\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223493 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysctl-conf\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223505 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-host\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223519 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qmj\" (UniqueName: \"kubernetes.io/projected/4a270284-98b1-4ec0-8c1a-48b7217f3754-kube-api-access-45qmj\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223535 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpj2\" (UniqueName: \"kubernetes.io/projected/9e01ea33-3313-47a6-a2b3-93a28babfdbd-kube-api-access-6kpj2\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223553 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b53d14f6-bfe9-4d19-ad46-d16aea78249c-hosts-file\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223568 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-cnibin\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223605 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-modprobe-d\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223666 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-cnibin\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223676 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysconfig\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.224231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223705 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-systemd\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223730 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223756 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e01ea33-3313-47a6-a2b3-93a28babfdbd-host-slash\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223777 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-multus-certs\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223785 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8afaa9e7-52cc-4050-9a84-03208d9d39c2-cni-binary-copy\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223804 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtv7\" (UniqueName: \"kubernetes.io/projected/8afaa9e7-52cc-4050-9a84-03208d9d39c2-kube-api-access-chtv7\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223828 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-kubernetes\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223846 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223860 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-run-multus-certs\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223889 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b581286c-01e6-45a0-a760-5bb689a8b8e3-kubelet-config\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223913 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-daemon-config\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223917 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223968 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-cni-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.223985 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-socket-dir-parent\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.224607 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224005 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-kubelet\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224021 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-hostroot\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224025 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-cni-dir\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224038 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224053 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-host-var-lib-kubelet\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224060 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-multus-socket-dir-parent\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224061 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224104 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8afaa9e7-52cc-4050-9a84-03208d9d39c2-hostroot\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224245 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53d14f6-bfe9-4d19-ad46-d16aea78249c-tmp-dir\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.225006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.224325 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-serviceca\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.228773 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.228752 2543 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 00:55:37.231715 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.231690 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6tp\" (UniqueName: \"kubernetes.io/projected/b53d14f6-bfe9-4d19-ad46-d16aea78249c-kube-api-access-2t6tp\") pod \"node-resolver-klfjr\" (UID: \"b53d14f6-bfe9-4d19-ad46-d16aea78249c\") " pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.231806 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.231734 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgg8\" (UniqueName: \"kubernetes.io/projected/0b9f359d-cb28-4cf9-90ce-248aa8fbd878-kube-api-access-5zgg8\") pod \"node-ca-s55gv\" (UID: \"0b9f359d-cb28-4cf9-90ce-248aa8fbd878\") " pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.231984 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.231970 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtv7\" (UniqueName: \"kubernetes.io/projected/8afaa9e7-52cc-4050-9a84-03208d9d39c2-kube-api-access-chtv7\") pod \"multus-9qmvg\" (UID: \"8afaa9e7-52cc-4050-9a84-03208d9d39c2\") " pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.243731 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.243715 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-fqlk7"] Jul 09 00:55:37.247611 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.247598 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.250324 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.250307 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 00:55:37.250734 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.250719 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 00:55:37.250971 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.250955 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 00:55:37.250971 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.250969 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 00:55:37.251074 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.250974 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 00:55:37.251074 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.250975 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bp4xs\"" Jul 09 00:55:37.251197 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.251185 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 00:55:37.324266 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324247 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.324370 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324274 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-socket-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.324370 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324290 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-systemd-units\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.324370 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324307 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysctl-d\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.324370 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324321 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-run-netns\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.324370 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324335 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cnibin\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.324370 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324350 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb98w\" (UniqueName: \"kubernetes.io/projected/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-kube-api-access-gb98w\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324388 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-etc-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324416 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cnibin\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324416 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-socket-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324444 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysctl-d\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324458 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-node-log\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324481 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c95d9938-3b58-44fd-a335-b743e9772e5e-agent-certs\") pod \"konnectivity-agent-649tx\" (UID: \"c95d9938-3b58-44fd-a335-b743e9772e5e\") " pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324501 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-os-release\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324554 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-cni-netd\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.324593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324585 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324607 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-os-release\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324610 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-var-lib-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324682 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324721 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-env-overrides\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324751 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-device-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324772 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324790 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-run\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324814 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-tuned\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324837 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b581286c-01e6-45a0-a760-5bb689a8b8e3-dbus\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324846 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-device-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324884 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-run\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324900 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b581286c-01e6-45a0-a760-5bb689a8b8e3-dbus\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324917 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-slash\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.325032 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324943 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-sys\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324964 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-lib-modules\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.324985 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-ovn\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325008 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17a639c0-fa11-4d1a-8d29-23066e726b22-ovn-node-metrics-cert\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325028 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-sys\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325034 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-sys-fs\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325061 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-system-cni-dir\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325085 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnj2\" (UniqueName: \"kubernetes.io/projected/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-kube-api-access-tdnj2\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325103 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-sys-fs\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325107 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325108 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-kubelet\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325034 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325156 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-lib-modules\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325162 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-system-cni-dir\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.325578 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325244 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-cni-bin\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325276 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5fk\" (UniqueName: \"kubernetes.io/projected/17a639c0-fa11-4d1a-8d29-23066e726b22-kube-api-access-vz5fk\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325316 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysctl-conf\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325343 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-host\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325367 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qmj\" (UniqueName: \"kubernetes.io/projected/4a270284-98b1-4ec0-8c1a-48b7217f3754-kube-api-access-45qmj\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325389 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpj2\" (UniqueName: \"kubernetes.io/projected/9e01ea33-3313-47a6-a2b3-93a28babfdbd-kube-api-access-6kpj2\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325405 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-host\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325414 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-ovnkube-script-lib\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325441 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-modprobe-d\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325444 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysctl-conf\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325511 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysconfig\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325466 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-sysconfig\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325566 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-modprobe-d\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325570 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-systemd\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325603 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325607 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-systemd\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325626 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e01ea33-3313-47a6-a2b3-93a28babfdbd-host-slash\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325672 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-kubernetes\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325688 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e01ea33-3313-47a6-a2b3-93a28babfdbd-host-slash\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325697 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325728 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-kubernetes\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325729 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b581286c-01e6-45a0-a760-5bb689a8b8e3-kubelet-config\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325756 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325772 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-systemd\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325792 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325806 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325820 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-log-socket\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325820 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b581286c-01e6-45a0-a760-5bb689a8b8e3-kubelet-config\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.326897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.325837 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-ovnkube-config\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.325902 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.325984 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret podName:b581286c-01e6-45a0-a760-5bb689a8b8e3 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:37.825944999 +0000 UTC m=+2.103783650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret") pod "global-pull-secret-syncer-thjzt" (UID: "b581286c-01e6-45a0-a760-5bb689a8b8e3") : object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326006 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c95d9938-3b58-44fd-a335-b743e9772e5e-konnectivity-ca\") pod \"konnectivity-agent-649tx\" (UID: \"c95d9938-3b58-44fd-a335-b743e9772e5e\") " pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326021 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326034 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326066 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-registration-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326092 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9e01ea33-3313-47a6-a2b3-93a28babfdbd-iptables-alerter-script\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326106 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326116 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-var-lib-kubelet\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326158 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94802df-4092-42e6-ac8c-c0408a4b415a-var-lib-kubelet\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326164 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94802df-4092-42e6-ac8c-c0408a4b415a-tmp\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326196 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a270284-98b1-4ec0-8c1a-48b7217f3754-registration-dir\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326272 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.327401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326301 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvhp\" (UniqueName: \"kubernetes.io/projected/f94802df-4092-42e6-ac8c-c0408a4b415a-kube-api-access-lzvhp\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326295 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326335 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326362 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.326421 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.326472 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs podName:c541c940-d9b8-4cb0-ac38-5abdc3e112c8 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:37.82645648 +0000 UTC m=+2.104295135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs") pod "network-metrics-daemon-69lnl" (UID: "c541c940-d9b8-4cb0-ac38-5abdc3e112c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326842 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9e01ea33-3313-47a6-a2b3-93a28babfdbd-iptables-alerter-script\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.326990 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c95d9938-3b58-44fd-a335-b743e9772e5e-konnectivity-ca\") pod \"konnectivity-agent-649tx\" (UID: \"c95d9938-3b58-44fd-a335-b743e9772e5e\") " pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.327051 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c95d9938-3b58-44fd-a335-b743e9772e5e-agent-certs\") pod \"konnectivity-agent-649tx\" (UID: \"c95d9938-3b58-44fd-a335-b743e9772e5e\") " pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.327833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.327171 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94802df-4092-42e6-ac8c-c0408a4b415a-etc-tuned\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.328097 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.328032 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94802df-4092-42e6-ac8c-c0408a4b415a-tmp\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.331405 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.331387 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 00:55:37.331493 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.331409 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 00:55:37.331493 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.331423 2543 projected.go:196] Error preparing data for projected volume kube-api-access-zrhbb for pod openshift-network-diagnostics/network-check-target-2mpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:37.331493 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.331481 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb podName:864bf335-d70a-4510-a93f-cbfb67948f9a nodeName:}" failed. No retries permitted until 2026-07-09 00:55:37.831466346 +0000 UTC m=+2.109304987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zrhbb" (UniqueName: "kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb") pod "network-check-target-2mpcm" (UID: "864bf335-d70a-4510-a93f-cbfb67948f9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:37.332725 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.332701 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb98w\" (UniqueName: \"kubernetes.io/projected/2a0c7205-318e-465c-bd0f-1e4734d2b8e5-kube-api-access-gb98w\") pod \"multus-additional-cni-plugins-wjb6p\" (UID: \"2a0c7205-318e-465c-bd0f-1e4734d2b8e5\") " pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.333343 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.333323 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qmj\" (UniqueName: \"kubernetes.io/projected/4a270284-98b1-4ec0-8c1a-48b7217f3754-kube-api-access-45qmj\") pod \"aws-ebs-csi-driver-node-k2srz\" (UID: \"4a270284-98b1-4ec0-8c1a-48b7217f3754\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.333817 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.333793 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvhp\" (UniqueName: \"kubernetes.io/projected/f94802df-4092-42e6-ac8c-c0408a4b415a-kube-api-access-lzvhp\") pod \"tuned-ghvhc\" (UID: \"f94802df-4092-42e6-ac8c-c0408a4b415a\") " pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.334006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.333988 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpj2\" (UniqueName: \"kubernetes.io/projected/9e01ea33-3313-47a6-a2b3-93a28babfdbd-kube-api-access-6kpj2\") pod \"iptables-alerter-dtxxn\" (UID: \"9e01ea33-3313-47a6-a2b3-93a28babfdbd\") " pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.334241 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.334226 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnj2\" (UniqueName: \"kubernetes.io/projected/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-kube-api-access-tdnj2\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:37.358528 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.358510 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.359586 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.359572 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Jul 09 00:55:37.427360 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427338 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-systemd-units\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427472 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427372 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-run-netns\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427472 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427396 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-etc-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427472 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427418 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-node-log\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427472 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427445 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-cni-netd\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427472 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427448 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-systemd-units\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427469 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-run-netns\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427485 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-node-log\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427488 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-cni-netd\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427475 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-etc-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427530 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-var-lib-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427548 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427565 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-env-overrides\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427582 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-slash\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427605 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-ovn\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427614 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427626 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17a639c0-fa11-4d1a-8d29-23066e726b22-ovn-node-metrics-cert\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427636 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-slash\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427666 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-var-lib-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.427689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427674 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-ovn\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427680 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-kubelet\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427699 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-cni-bin\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427715 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5fk\" (UniqueName: \"kubernetes.io/projected/17a639c0-fa11-4d1a-8d29-23066e726b22-kube-api-access-vz5fk\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427733 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-ovnkube-script-lib\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427740 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-kubelet\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427778 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-cni-bin\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427787 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-systemd\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427817 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-log-socket\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427840 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-ovnkube-config\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427817 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-systemd\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427849 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-log-socket\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427893 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427921 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427955 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-run-openvswitch\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428248 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.427981 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17a639c0-fa11-4d1a-8d29-23066e726b22-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428711 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.428063 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-env-overrides\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428711 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.428174 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-ovnkube-script-lib\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.428711 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.428263 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17a639c0-fa11-4d1a-8d29-23066e726b22-ovnkube-config\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.429496 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.429481 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17a639c0-fa11-4d1a-8d29-23066e726b22-ovn-node-metrics-cert\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.435997 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.435976 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5fk\" (UniqueName: \"kubernetes.io/projected/17a639c0-fa11-4d1a-8d29-23066e726b22-kube-api-access-vz5fk\") pod \"ovnkube-node-fqlk7\" (UID: \"17a639c0-fa11-4d1a-8d29-23066e726b22\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.436825 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.436812 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klfjr" Jul 09 00:55:37.451436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.451420 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s55gv" Jul 09 00:55:37.471053 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.471035 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9qmvg" Jul 09 00:55:37.486192 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.486174 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" Jul 09 00:55:37.499678 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.499664 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dtxxn" Jul 09 00:55:37.515198 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.515183 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:37.528733 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.528710 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" Jul 09 00:55:37.538271 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.538256 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" Jul 09 00:55:37.565052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.565036 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:55:37.831302 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.831232 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:37.831302 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.831275 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:37.831510 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.831381 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:37.831510 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.831399 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:37.831510 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.831442 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs podName:c541c940-d9b8-4cb0-ac38-5abdc3e112c8 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:38.831423891 +0000 UTC m=+3.109262541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs") pod "network-metrics-daemon-69lnl" (UID: "c541c940-d9b8-4cb0-ac38-5abdc3e112c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:37.831510 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.831463 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret podName:b581286c-01e6-45a0-a760-5bb689a8b8e3 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:38.831452009 +0000 UTC m=+3.109290649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret") pod "global-pull-secret-syncer-thjzt" (UID: "b581286c-01e6-45a0-a760-5bb689a8b8e3") : object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:37.931982 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.931955 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:37.932186 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.932109 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 00:55:37.932186 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.932128 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 00:55:37.932186 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.932137 2543 projected.go:196] Error preparing data for projected volume kube-api-access-zrhbb for pod openshift-network-diagnostics/network-check-target-2mpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:37.932186 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:37.932186 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb podName:864bf335-d70a-4510-a93f-cbfb67948f9a nodeName:}" failed. No retries permitted until 2026-07-09 00:55:38.932171399 +0000 UTC m=+3.210010027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrhbb" (UniqueName: "kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb") pod "network-check-target-2mpcm" (UID: "864bf335-d70a-4510-a93f-cbfb67948f9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:37.981680 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:37.981635 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9f359d_cb28_4cf9_90ce_248aa8fbd878.slice/crio-0cd8b990e436b95a90eb966ccaf43c2099ef2dc0433e1bb58badf206f8ed26dc WatchSource:0}: Error finding container 0cd8b990e436b95a90eb966ccaf43c2099ef2dc0433e1bb58badf206f8ed26dc: Status 404 returned error can't find the container with id 0cd8b990e436b95a90eb966ccaf43c2099ef2dc0433e1bb58badf206f8ed26dc Jul 09 00:55:37.983027 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:37.983000 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a639c0_fa11_4d1a_8d29_23066e726b22.slice/crio-68e35ccd50dbe0e0367d569de0870c12a2f3ce3cd55e746ca166ae8c5bf1f6dd WatchSource:0}: Error finding container 68e35ccd50dbe0e0367d569de0870c12a2f3ce3cd55e746ca166ae8c5bf1f6dd: Status 404 returned error can't find the container with id 68e35ccd50dbe0e0367d569de0870c12a2f3ce3cd55e746ca166ae8c5bf1f6dd Jul 09 00:55:37.985453 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:37.985430 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd838a4b982cf3dfd5cfb41e87f7285.slice/crio-aded70aa0dc0959e02a6d95bee412466a2b476201ca885348fc367ac8f5431a4 WatchSource:0}: Error finding container aded70aa0dc0959e02a6d95bee412466a2b476201ca885348fc367ac8f5431a4: Status 404 returned error can't find the container with id aded70aa0dc0959e02a6d95bee412466a2b476201ca885348fc367ac8f5431a4 Jul 09 00:55:37.986429 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:37.986411 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 00:55:37.987296 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:37.987272 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53d14f6_bfe9_4d19_ad46_d16aea78249c.slice/crio-c09ed1e7a4219d82d0c2ff9199f7f58b8cec3e92bce604fa20cc2354dd857e99 WatchSource:0}: Error finding container c09ed1e7a4219d82d0c2ff9199f7f58b8cec3e92bce604fa20cc2354dd857e99: Status 404 returned error can't find the container with id c09ed1e7a4219d82d0c2ff9199f7f58b8cec3e92bce604fa20cc2354dd857e99 Jul 09 00:55:37.990248 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:37.990225 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a270284_98b1_4ec0_8c1a_48b7217f3754.slice/crio-f9c3c769818e3083c5c1827043a62320b32d2e6aa7b94a144420b5aa6d4658dc WatchSource:0}: Error finding container f9c3c769818e3083c5c1827043a62320b32d2e6aa7b94a144420b5aa6d4658dc: Status 404 returned error can't find the container with id f9c3c769818e3083c5c1827043a62320b32d2e6aa7b94a144420b5aa6d4658dc Jul 09 00:55:38.016402 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:38.016380 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afaa9e7_52cc_4050_9a84_03208d9d39c2.slice/crio-6909623b0c5fee248f2bd0ad633b331e49e7ab61b3f8433e0a5e5866aca13c21 WatchSource:0}: Error finding container 6909623b0c5fee248f2bd0ad633b331e49e7ab61b3f8433e0a5e5866aca13c21: Status 404 returned error can't find the container with id 6909623b0c5fee248f2bd0ad633b331e49e7ab61b3f8433e0a5e5866aca13c21 Jul 09 00:55:38.046856 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:38.046835 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e01ea33_3313_47a6_a2b3_93a28babfdbd.slice/crio-ee954f3e49eba3d54bf010bd81544479402775d723d1789e4bc05760bd0bf1ef WatchSource:0}: Error finding container ee954f3e49eba3d54bf010bd81544479402775d723d1789e4bc05760bd0bf1ef: Status 404 returned error can't find the container with id ee954f3e49eba3d54bf010bd81544479402775d723d1789e4bc05760bd0bf1ef Jul 09 00:55:38.053869 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:38.053845 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95d9938_3b58_44fd_a335_b743e9772e5e.slice/crio-1a8ad7b3199be5e31974eda5c92475783ef441f37a3473163847ce537230db8c WatchSource:0}: Error finding container 1a8ad7b3199be5e31974eda5c92475783ef441f37a3473163847ce537230db8c: Status 404 returned error can't find the container with id 1a8ad7b3199be5e31974eda5c92475783ef441f37a3473163847ce537230db8c Jul 09 00:55:38.063029 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:38.063013 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c46fe0b45721f50ed4b3952298077b2.slice/crio-3a1e8a00edd21cbd2e1c034033f28ce0b96864c97983dbe03a6de89bec649009 WatchSource:0}: Error finding container 3a1e8a00edd21cbd2e1c034033f28ce0b96864c97983dbe03a6de89bec649009: Status 404 returned error can't find the container with id 3a1e8a00edd21cbd2e1c034033f28ce0b96864c97983dbe03a6de89bec649009 Jul 09 00:55:38.089711 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:38.089692 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0c7205_318e_465c_bd0f_1e4734d2b8e5.slice/crio-a042d98b8deb6c21d185d1e20be5493cc00a58c8228624f4e283e2f708f5eb33 WatchSource:0}: Error finding container a042d98b8deb6c21d185d1e20be5493cc00a58c8228624f4e283e2f708f5eb33: Status 404 returned error can't find the container with id a042d98b8deb6c21d185d1e20be5493cc00a58c8228624f4e283e2f708f5eb33 Jul 09 00:55:38.118238 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:55:38.118219 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94802df_4092_42e6_ac8c_c0408a4b415a.slice/crio-b779cb5aef8c05e23f32b0124a5f50baf80ebbbd3c54b453bde90312cc8e57f7 WatchSource:0}: Error finding container b779cb5aef8c05e23f32b0124a5f50baf80ebbbd3c54b453bde90312cc8e57f7: Status 404 returned error can't find the container with id b779cb5aef8c05e23f32b0124a5f50baf80ebbbd3c54b453bde90312cc8e57f7 Jul 09 00:55:38.119839 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.119822 2543 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-ghvhc_openshift-cluster-node-tuning-operator(f94802df-4092-42e6-ac8c-c0408a4b415a): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 09 00:55:38.119889 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.119853 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" podUID="f94802df-4092-42e6-ac8c-c0408a4b415a" Jul 09 00:55:38.166015 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.165966 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 00:50:37 +0000 UTC" deadline="2028-03-11 15:00:13.360316906 +0000 UTC" Jul 09 00:55:38.166015 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.166012 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14678h4m35.19430947s" Jul 09 00:55:38.373225 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.373157 2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 00:55:38.602448 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.602137 2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 00:55:38.840755 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.840334 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:38.840755 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.840419 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:38.840755 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.840546 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:38.840755 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.840605 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret podName:b581286c-01e6-45a0-a760-5bb689a8b8e3 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:40.840584484 +0000 UTC m=+5.118423114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret") pod "global-pull-secret-syncer-thjzt" (UID: "b581286c-01e6-45a0-a760-5bb689a8b8e3") : object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:38.840755 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.840692 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:38.840755 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.840726 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs podName:c541c940-d9b8-4cb0-ac38-5abdc3e112c8 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:40.840714623 +0000 UTC m=+5.118553254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs") pod "network-metrics-daemon-69lnl" (UID: "c541c940-d9b8-4cb0-ac38-5abdc3e112c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:38.923172 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.922752 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:38.923172 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.922881 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:38.923172 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.922961 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:38.923172 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.923038 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:38.923782 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.923563 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:38.923782 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.923676 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:38.944338 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.943796 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-649tx" event={"ID":"c95d9938-3b58-44fd-a335-b743e9772e5e","Type":"ContainerStarted","Data":"1a8ad7b3199be5e31974eda5c92475783ef441f37a3473163847ce537230db8c"} Jul 09 00:55:38.944338 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.944075 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:38.944338 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.944239 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 00:55:38.944338 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.944258 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 00:55:38.944338 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.944271 2543 projected.go:196] Error preparing data for projected volume kube-api-access-zrhbb for pod openshift-network-diagnostics/network-check-target-2mpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:38.944338 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:38.944316 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb podName:864bf335-d70a-4510-a93f-cbfb67948f9a nodeName:}" failed. No retries permitted until 2026-07-09 00:55:40.944299883 +0000 UTC m=+5.222138526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrhbb" (UniqueName: "kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb") pod "network-check-target-2mpcm" (UID: "864bf335-d70a-4510-a93f-cbfb67948f9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:38.954872 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.954842 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dtxxn" event={"ID":"9e01ea33-3313-47a6-a2b3-93a28babfdbd","Type":"ContainerStarted","Data":"ee954f3e49eba3d54bf010bd81544479402775d723d1789e4bc05760bd0bf1ef"} Jul 09 00:55:38.971584 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.971559 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9qmvg" event={"ID":"8afaa9e7-52cc-4050-9a84-03208d9d39c2","Type":"ContainerStarted","Data":"6909623b0c5fee248f2bd0ad633b331e49e7ab61b3f8433e0a5e5866aca13c21"} Jul 09 00:55:38.996514 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:38.996488 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" event={"ID":"4a270284-98b1-4ec0-8c1a-48b7217f3754","Type":"ContainerStarted","Data":"f9c3c769818e3083c5c1827043a62320b32d2e6aa7b94a144420b5aa6d4658dc"} Jul 09 00:55:39.010144 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.010119 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klfjr" event={"ID":"b53d14f6-bfe9-4d19-ad46-d16aea78249c","Type":"ContainerStarted","Data":"c09ed1e7a4219d82d0c2ff9199f7f58b8cec3e92bce604fa20cc2354dd857e99"} Jul 09 00:55:39.012887 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.012845 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s55gv" event={"ID":"0b9f359d-cb28-4cf9-90ce-248aa8fbd878","Type":"ContainerStarted","Data":"0cd8b990e436b95a90eb966ccaf43c2099ef2dc0433e1bb58badf206f8ed26dc"} Jul 09 00:55:39.038153 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.038128 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" event={"ID":"1c46fe0b45721f50ed4b3952298077b2","Type":"ContainerStarted","Data":"3a1e8a00edd21cbd2e1c034033f28ce0b96864c97983dbe03a6de89bec649009"} Jul 09 00:55:39.061037 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.061014 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" event={"ID":"3fd838a4b982cf3dfd5cfb41e87f7285","Type":"ContainerStarted","Data":"aded70aa0dc0959e02a6d95bee412466a2b476201ca885348fc367ac8f5431a4"} Jul 09 00:55:39.077091 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.076982 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"68e35ccd50dbe0e0367d569de0870c12a2f3ce3cd55e746ca166ae8c5bf1f6dd"} Jul 09 00:55:39.079158 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.079138 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" event={"ID":"f94802df-4092-42e6-ac8c-c0408a4b415a","Type":"ContainerStarted","Data":"b779cb5aef8c05e23f32b0124a5f50baf80ebbbd3c54b453bde90312cc8e57f7"} Jul 09 00:55:39.089499 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:39.089472 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" podUID="f94802df-4092-42e6-ac8c-c0408a4b415a" Jul 09 00:55:39.099108 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.099052 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerStarted","Data":"a042d98b8deb6c21d185d1e20be5493cc00a58c8228624f4e283e2f708f5eb33"} Jul 09 00:55:39.166893 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.166841 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 00:50:37 +0000 UTC" deadline="2028-01-23 19:47:58.69038755 +0000 UTC" Jul 09 00:55:39.166893 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.166891 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13530h52m19.523499787s" Jul 09 00:55:39.420771 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:39.420526 2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 00:55:40.861516 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:40.861202 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:40.861516 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:40.861466 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:40.862032 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.861616 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:40.862032 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.861687 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs podName:c541c940-d9b8-4cb0-ac38-5abdc3e112c8 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:44.861667747 +0000 UTC m=+9.139506390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs") pod "network-metrics-daemon-69lnl" (UID: "c541c940-d9b8-4cb0-ac38-5abdc3e112c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:40.862032 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.861758 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:40.862032 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.861790 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret podName:b581286c-01e6-45a0-a760-5bb689a8b8e3 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:44.861778775 +0000 UTC m=+9.139617419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret") pod "global-pull-secret-syncer-thjzt" (UID: "b581286c-01e6-45a0-a760-5bb689a8b8e3") : object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:40.923782 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:40.923747 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:40.923939 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.923886 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:40.924002 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:40.923958 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:40.924053 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.924028 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:40.924108 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:40.924090 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:40.924176 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.924153 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:40.962735 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:40.962705 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:40.962922 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.962905 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 00:55:40.962992 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.962932 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 00:55:40.962992 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.962944 2543 projected.go:196] Error preparing data for projected volume kube-api-access-zrhbb for pod openshift-network-diagnostics/network-check-target-2mpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:40.963092 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:40.962994 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb podName:864bf335-d70a-4510-a93f-cbfb67948f9a nodeName:}" failed. No retries permitted until 2026-07-09 00:55:44.962976827 +0000 UTC m=+9.240815461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrhbb" (UniqueName: "kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb") pod "network-check-target-2mpcm" (UID: "864bf335-d70a-4510-a93f-cbfb67948f9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:42.922935 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:42.922904 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:42.923383 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:42.923025 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:42.923383 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:42.922904 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:42.923383 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:42.923016 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:42.923383 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:42.923137 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:42.923383 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:42.923224 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:44.889635 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:44.889593 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:44.890050 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:44.889657 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:44.890050 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.889763 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:44.890050 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.889841 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret podName:b581286c-01e6-45a0-a760-5bb689a8b8e3 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:52.889820716 +0000 UTC m=+17.167659348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret") pod "global-pull-secret-syncer-thjzt" (UID: "b581286c-01e6-45a0-a760-5bb689a8b8e3") : object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:44.890050 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.889769 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:44.890050 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.889891 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs podName:c541c940-d9b8-4cb0-ac38-5abdc3e112c8 nodeName:}" failed. No retries permitted until 2026-07-09 00:55:52.889881177 +0000 UTC m=+17.167719812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs") pod "network-metrics-daemon-69lnl" (UID: "c541c940-d9b8-4cb0-ac38-5abdc3e112c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:44.922917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:44.922822 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:44.922917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:44.922833 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:44.923123 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:44.922824 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:44.923123 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.922945 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:44.923123 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.923021 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:44.923123 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.923097 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:44.990366 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:44.990332 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:44.990579 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.990554 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 00:55:44.990579 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.990580 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 00:55:44.990771 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.990594 2543 projected.go:196] Error preparing data for projected volume kube-api-access-zrhbb for pod openshift-network-diagnostics/network-check-target-2mpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:44.990771 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:44.990686 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb podName:864bf335-d70a-4510-a93f-cbfb67948f9a nodeName:}" failed. No retries permitted until 2026-07-09 00:55:52.990665273 +0000 UTC m=+17.268503908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrhbb" (UniqueName: "kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb") pod "network-check-target-2mpcm" (UID: "864bf335-d70a-4510-a93f-cbfb67948f9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:46.924105 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:46.923902 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:46.924105 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:46.923933 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:46.924105 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:46.924044 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:46.924681 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:46.924122 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:46.924681 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:46.924181 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:46.924681 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:46.924251 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:47.126836 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.126763 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klfjr" event={"ID":"b53d14f6-bfe9-4d19-ad46-d16aea78249c","Type":"ContainerStarted","Data":"5e6b931f2a3ce2283d8815b6ca008af2d0e6632e064e87298763159237a470a4"} Jul 09 00:55:47.128200 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.128158 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s55gv" event={"ID":"0b9f359d-cb28-4cf9-90ce-248aa8fbd878","Type":"ContainerStarted","Data":"60ad8513bc50b40e58fa8636ed7544b0a4abc0399d4895f1cd7cff47acbc52b9"} Jul 09 00:55:47.129495 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.129470 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" event={"ID":"1c46fe0b45721f50ed4b3952298077b2","Type":"ContainerStarted","Data":"08a1987d97b622f6fe6446ea1d1590a410bca314636f1a48bffbdd337ef41bcc"} Jul 09 00:55:47.130862 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.130837 2543 generic.go:356] "Generic (PLEG): container finished" podID="3fd838a4b982cf3dfd5cfb41e87f7285" containerID="ceabd159e697b4585d73bba2f6f37e5a96e5d955bb9533ce8bda5113f3aac9cb" exitCode=0 Jul 09 00:55:47.130964 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.130925 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" event={"ID":"3fd838a4b982cf3dfd5cfb41e87f7285","Type":"ContainerDied","Data":"ceabd159e697b4585d73bba2f6f37e5a96e5d955bb9533ce8bda5113f3aac9cb"} Jul 09 00:55:47.132687 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.132638 2543 generic.go:356] "Generic (PLEG): container finished" podID="2a0c7205-318e-465c-bd0f-1e4734d2b8e5" containerID="ac4dc8c3760f20c7f78fa2820506d1c6e6b386c5ea4ac84f0a45757aa2c06833" exitCode=0 Jul 09 00:55:47.132791 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.132735 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerDied","Data":"ac4dc8c3760f20c7f78fa2820506d1c6e6b386c5ea4ac84f0a45757aa2c06833"} Jul 09 00:55:47.134291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.134266 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-649tx" event={"ID":"c95d9938-3b58-44fd-a335-b743e9772e5e","Type":"ContainerStarted","Data":"42ff461c58be7e615bf47e878000cf3d38def5fc7fdaf7f5f67512d15781abdb"} Jul 09 00:55:47.136064 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.136042 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" event={"ID":"4a270284-98b1-4ec0-8c1a-48b7217f3754","Type":"ContainerStarted","Data":"e9aa9d2e7bc907fe097a133fb7e2ac53ab0f1c5f5bd24d6ab4d0c8c94c490bb4"} Jul 09 00:55:47.144349 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.144305 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-klfjr" podStartSLOduration=2.661187926 podStartE2EDuration="11.144290791s" totalImagesPullingTime="8.483102865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:55:47.144027118 +0000 UTC m=+11.421865770" watchObservedRunningTime="2026-07-09 00:55:47.144290791 +0000 UTC m=+11.422129516" Jul 09 00:55:47.184325 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.184277 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" podStartSLOduration=10.184257369000001 podStartE2EDuration="10.184257369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:55:47.170589036 +0000 UTC m=+11.448427689" watchObservedRunningTime="2026-07-09 00:55:47.184257369 +0000 UTC m=+11.462096101" Jul 09 00:55:47.184454 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.184396 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s55gv" podStartSLOduration=2.700502498 podStartE2EDuration="11.184389495s" totalImagesPullingTime="8.483886997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:55:47.183413986 +0000 UTC m=+11.461252638" watchObservedRunningTime="2026-07-09 00:55:47.184389495 +0000 UTC m=+11.462228148" Jul 09 00:55:47.224637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.224467 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-649tx" podStartSLOduration=1.8087512380000001 podStartE2EDuration="10.224452369s" totalImagesPullingTime="8.415701131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:55:47.223874199 +0000 UTC m=+11.501712875" watchObservedRunningTime="2026-07-09 00:55:47.224452369 +0000 UTC m=+11.502291020" Jul 09 00:55:47.567151 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.567117 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:47.968551 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:47.968520 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:48.140080 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.139999 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dtxxn" event={"ID":"9e01ea33-3313-47a6-a2b3-93a28babfdbd","Type":"ContainerStarted","Data":"e77124a89ca5a5652f9926965713351a84ee30b93a533dfa80a6e7df83a78da6"} Jul 09 00:55:48.142101 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.142074 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" event={"ID":"3fd838a4b982cf3dfd5cfb41e87f7285","Type":"ContainerStarted","Data":"25f0996087ef01f015e7ee2c70bb6fa9ed15403420f18f07bdf715e1167878d1"} Jul 09 00:55:48.153730 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.153654 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dtxxn" podStartSLOduration=2.731920333 podStartE2EDuration="11.153621486s" totalImagesPullingTime="8.421701153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:55:48.153473357 +0000 UTC m=+12.431312008" watchObservedRunningTime="2026-07-09 00:55:48.153621486 +0000 UTC m=+12.431460140" Jul 09 00:55:48.162493 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.162462 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:48.168177 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.168131 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" podStartSLOduration=11.168113136 podStartE2EDuration="11.168113136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:55:48.16758804 +0000 UTC m=+12.445426691" watchObservedRunningTime="2026-07-09 00:55:48.168113136 +0000 UTC m=+12.445951789" Jul 09 00:55:48.246349 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.246320 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:48.566220 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.566182 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:48.764799 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.764731 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:48.865295 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.865220 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:48.922816 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.922735 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:48.922816 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.922765 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:48.923036 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:48.922735 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:48.923036 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:48.922860 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:48.923036 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:48.922938 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:48.923186 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:48.923043 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:49.364054 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:49.364009 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:49.564426 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:49.564394 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:49.662966 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:49.662883 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:49.965917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:49.965888 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:50.564920 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:50.564883 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:50.617365 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:50.617330 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:50.922984 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:50.922913 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:50.922984 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:50.922914 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:50.923189 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:50.922914 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:50.923189 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:50.923037 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:50.923189 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:50.923150 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:50.923339 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:50.923266 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:51.226556 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:51.226513 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:51.361668 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:51.361619 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:51.683295 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:51.683219 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:51.683988 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:51.683952 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-649tx" Jul 09 00:55:51.762263 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:51.762232 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:51.960277 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:51.960250 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:52.563157 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:52.563124 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:52.923075 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:52.922998 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:52.923075 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:52.923030 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:52.923075 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:52.923005 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:52.923668 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:52.923621 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:52.923824 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:52.923796 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:52.923946 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:52.923925 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:52.950108 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:52.950076 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:52.950220 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:52.950128 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:52.950274 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:52.950221 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:52.950274 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:52.950230 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:52.950351 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:52.950275 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs podName:c541c940-d9b8-4cb0-ac38-5abdc3e112c8 nodeName:}" failed. No retries permitted until 2026-07-09 00:56:08.950258375 +0000 UTC m=+33.228097009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs") pod "network-metrics-daemon-69lnl" (UID: "c541c940-d9b8-4cb0-ac38-5abdc3e112c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:55:52.950351 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:52.950290 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret podName:b581286c-01e6-45a0-a760-5bb689a8b8e3 nodeName:}" failed. No retries permitted until 2026-07-09 00:56:08.950283726 +0000 UTC m=+33.228122356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret") pod "global-pull-secret-syncer-thjzt" (UID: "b581286c-01e6-45a0-a760-5bb689a8b8e3") : object "kube-system"/"original-pull-secret" not registered Jul 09 00:55:52.963452 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:52.963422 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:53.050450 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:53.050416 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:53.050618 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:53.050577 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 00:55:53.050618 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:53.050604 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 00:55:53.050618 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:53.050616 2543 projected.go:196] Error preparing data for projected volume kube-api-access-zrhbb for pod openshift-network-diagnostics/network-check-target-2mpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:53.050789 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:53.050690 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb podName:864bf335-d70a-4510-a93f-cbfb67948f9a nodeName:}" failed. No retries permitted until 2026-07-09 00:56:09.050671062 +0000 UTC m=+33.328509698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrhbb" (UniqueName: "kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb") pod "network-check-target-2mpcm" (UID: "864bf335-d70a-4510-a93f-cbfb67948f9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:55:53.146747 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:53.146720 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:53.201462 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:53.201431 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:53.758334 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:53.758295 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:54.365462 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:54.365432 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:54.566432 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:54.566360 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:54.762461 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:54.762413 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:54.923202 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:54.922979 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:54.923202 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:54.923018 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:54.923202 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:54.922992 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:54.923202 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:54.923119 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:54.923202 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:54.923193 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:54.923717 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:54.923269 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:54.962072 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:54.962039 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:55.978287 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:55.978184 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:55:56.924130 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:56.924096 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:56.924291 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:56.924194 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:56.924368 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:56.924301 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:56.924426 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:56.924394 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:56.924478 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:56.924457 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:56.924595 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:56.924565 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:58.344014 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:58.343788 2543 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 00:55:58.823814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:58.823710 2543 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T00:55:58.344005869Z","UUID":"8d11f5b7-35ff-4f28-90a6-a0804adfd0f5","Handler":null,"Name":"","Endpoint":""} Jul 09 00:55:58.825992 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:58.825967 2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 00:55:58.826125 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:58.826012 2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 00:55:58.922666 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:58.922628 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:55:58.922825 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:58.922676 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:55:58.922825 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:58.922634 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:55:58.922825 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:58.922774 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:55:58.922963 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:58.922837 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:55:58.922963 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:55:58.922935 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:55:59.164612 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:59.164536 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" event={"ID":"4a270284-98b1-4ec0-8c1a-48b7217f3754","Type":"ContainerStarted","Data":"d03251880a8b5d83c5be1406e43757f776b2a319a155ea1f173e84b82f3ec20a"} Jul 09 00:55:59.166240 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:59.166213 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"cf7b38f0b6e976e1ba48d34f312cc03640f1e81b9cd4b4fae5e847f1330d9240"} Jul 09 00:55:59.168198 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:59.168172 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerStarted","Data":"c92fb7fd89a3cc108799316af2167f7c8eee267c75f5c34a43f8ea2a18798c92"} Jul 09 00:55:59.172622 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:59.172593 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9qmvg" event={"ID":"8afaa9e7-52cc-4050-9a84-03208d9d39c2","Type":"ContainerStarted","Data":"3f2acc76624b978448d0c80b0e796b6fad6baf1529e023c60bb935a1a3b7e11b"} Jul 09 00:55:59.203334 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:55:59.203291 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9qmvg" podStartSLOduration=3.103391475 podStartE2EDuration="23.203277833s" totalImagesPullingTime="20.099886358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:55:59.203117426 +0000 UTC m=+23.480956078" watchObservedRunningTime="2026-07-09 00:55:59.203277833 +0000 UTC m=+23.481116480" Jul 09 00:56:00.923446 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:00.923391 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:00.924048 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:00.923496 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:00.924048 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:00.923505 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:00.924048 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:00.923515 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:00.924048 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:00.923577 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:00.924048 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:00.923682 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:01.178828 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.178750 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" event={"ID":"f94802df-4092-42e6-ac8c-c0408a4b415a","Type":"ContainerStarted","Data":"4089ea15e599d28f7a16909cd07fd9f327ce2f20ddc6713f5ad88bdcf53720d8"} Jul 09 00:56:01.180628 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.180599 2543 generic.go:356] "Generic (PLEG): container finished" podID="2a0c7205-318e-465c-bd0f-1e4734d2b8e5" containerID="c92fb7fd89a3cc108799316af2167f7c8eee267c75f5c34a43f8ea2a18798c92" exitCode=0 Jul 09 00:56:01.180772 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.180674 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerDied","Data":"c92fb7fd89a3cc108799316af2167f7c8eee267c75f5c34a43f8ea2a18798c92"} Jul 09 00:56:01.184068 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.184023 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"7f938cea7bbd9235ba140082acf1a516864d8c7703e6c52ce596d8630e8bd620"} Jul 09 00:56:01.184178 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.184113 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"7012d9f360286b275f6524c6ec6c79eb7fe5c4e184cb0ae005d0808a73acf57e"} Jul 09 00:56:01.184178 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.184167 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"6d100c3200e08fe50ab511ac9b73a24c6f9a91e007dcb8a0afaefbce9531e42c"} Jul 09 00:56:01.184273 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.184212 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"3bb085fe25f499404ac76bfe9e8b930516990e4ce103fc5b845bd0c312a244f5"} Jul 09 00:56:01.184273 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.184240 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"860bac4b917d02104f9ebe74c31370b031b2cde92acb8b7c12a6bca80e6b51a9"} Jul 09 00:56:01.207851 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:01.207804 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ghvhc" podStartSLOduration=1.671933903 podStartE2EDuration="24.207789149s" totalImagesPullingTime="22.535855246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 00:56:01.207284319 +0000 UTC m=+25.485122972" watchObservedRunningTime="2026-07-09 00:56:01.207789149 +0000 UTC m=+25.485627800" Jul 09 00:56:02.187790 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:02.187754 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" event={"ID":"4a270284-98b1-4ec0-8c1a-48b7217f3754","Type":"ContainerStarted","Data":"a340252668636551644204110e5c52246391f3cdaf47f48674297b1a707178c0"} Jul 09 00:56:02.925451 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:02.925420 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:02.925451 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:02.925451 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:02.925668 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:02.925461 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:02.925668 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:02.925542 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:02.925668 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:02.925661 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:02.925815 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:02.925757 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:03.193156 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:03.193127 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"b0a36fc1ce36f32ef80a4f6afca841b4f3075b9d4c4d2b8328ad8e99d1f1b0f1"} Jul 09 00:56:03.194961 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:03.194934 2543 generic.go:356] "Generic (PLEG): container finished" podID="2a0c7205-318e-465c-bd0f-1e4734d2b8e5" containerID="b0586beccfa38dd2469e78ab6a2354659eb9efa9891475a7951fb78bcea30bbc" exitCode=0 Jul 09 00:56:03.195065 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:03.195004 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerDied","Data":"b0586beccfa38dd2469e78ab6a2354659eb9efa9891475a7951fb78bcea30bbc"} Jul 09 00:56:03.219555 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:03.218615 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2srz" podStartSLOduration=3.23885424 podStartE2EDuration="26.218598208s" totalImagesPullingTime="22.979743968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:02.206330589 +0000 UTC m=+26.484169239" watchObservedRunningTime="2026-07-09 00:56:03.218598208 +0000 UTC m=+27.496436862" Jul 09 00:56:04.925534 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:04.925508 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:04.925993 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:04.925669 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:04.925993 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:04.925706 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:04.925993 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:04.925731 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:04.925993 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:04.925813 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:04.925993 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:04.925876 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:05.200535 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:05.200405 2543 generic.go:356] "Generic (PLEG): container finished" podID="2a0c7205-318e-465c-bd0f-1e4734d2b8e5" containerID="ae9fbd5e2e05e81b0949d258e6fbc11fa01abe78d8ea0cbb32802d29e3b5f31a" exitCode=0 Jul 09 00:56:05.200656 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:05.200476 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerDied","Data":"ae9fbd5e2e05e81b0949d258e6fbc11fa01abe78d8ea0cbb32802d29e3b5f31a"} Jul 09 00:56:05.663233 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:05.663167 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:05.737014 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:05.736984 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:05.834456 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:05.834437 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:06.037584 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.035186 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:06.104041 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.104015 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:06.206365 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.206333 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" event={"ID":"17a639c0-fa11-4d1a-8d29-23066e726b22","Type":"ContainerStarted","Data":"a1dab72687ea2877355fb79ca87168932861c6011c4bf7bc60ef6343ff3e8b35"} Jul 09 00:56:06.206597 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.206573 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:56:06.222307 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.222290 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:56:06.235276 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.235240 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" podStartSLOduration=9.099483445 podStartE2EDuration="29.235229259s" totalImagesPullingTime="20.135745814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:06.235108169 +0000 UTC m=+30.512946820" watchObservedRunningTime="2026-07-09 00:56:06.235229259 +0000 UTC m=+30.513067911" Jul 09 00:56:06.887671 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.887621 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:06.924372 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.923929 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:06.924372 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.924014 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:06.924372 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:06.924028 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:06.924372 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:06.924134 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:06.924372 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:06.924198 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:06.924372 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:06.924293 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:07.201430 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.201391 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-thjzt"] Jul 09 00:56:07.204860 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.204835 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2mpcm"] Jul 09 00:56:07.205597 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.205572 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-69lnl"] Jul 09 00:56:07.209436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.209417 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:07.209552 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:07.209527 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:07.209620 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.209556 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:07.209620 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.209420 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:07.210069 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:07.210040 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:07.210223 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:07.210167 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:07.210296 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.210249 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:56:07.210356 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.210327 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:56:07.229721 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.229687 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:56:07.292061 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.292038 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-649tx" Jul 09 00:56:07.292188 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.292154 2543 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 00:56:07.292563 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.292544 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-649tx" Jul 09 00:56:07.350257 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:07.350235 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:08.706579 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.706545 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:08.809138 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.809104 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:08.872733 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.872697 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:08.923306 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.923281 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:08.923306 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.923293 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:08.923482 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:08.923401 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:08.923528 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:08.923479 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:08.923528 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.923509 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:08.923622 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:08.923602 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:08.969407 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.969347 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:08.969407 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:08.969382 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:08.969539 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:08.969460 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:56:08.969539 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:08.969475 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 00:56:08.969539 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:08.969508 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs podName:c541c940-d9b8-4cb0-ac38-5abdc3e112c8 nodeName:}" failed. No retries permitted until 2026-07-09 00:56:40.96949578 +0000 UTC m=+65.247334409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs") pod "network-metrics-daemon-69lnl" (UID: "c541c940-d9b8-4cb0-ac38-5abdc3e112c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 00:56:08.969539 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:08.969524 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret podName:b581286c-01e6-45a0-a760-5bb689a8b8e3 nodeName:}" failed. No retries permitted until 2026-07-09 00:56:40.969512842 +0000 UTC m=+65.247351470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret") pod "global-pull-secret-syncer-thjzt" (UID: "b581286c-01e6-45a0-a760-5bb689a8b8e3") : object "kube-system"/"original-pull-secret" not registered Jul 09 00:56:09.070177 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:09.070156 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:09.070280 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:09.070251 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 00:56:09.070280 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:09.070267 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 00:56:09.070280 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:09.070276 2543 projected.go:196] Error preparing data for projected volume kube-api-access-zrhbb for pod openshift-network-diagnostics/network-check-target-2mpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:56:09.070394 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:09.070311 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb podName:864bf335-d70a-4510-a93f-cbfb67948f9a nodeName:}" failed. No retries permitted until 2026-07-09 00:56:41.070301661 +0000 UTC m=+65.348140303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrhbb" (UniqueName: "kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb") pod "network-check-target-2mpcm" (UID: "864bf335-d70a-4510-a93f-cbfb67948f9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 00:56:09.952550 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:09.952390 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:10.044614 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:10.044577 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:10.146660 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:10.146619 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:10.242454 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:10.242423 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:10.400390 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:10.400359 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:10.923525 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:10.923497 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:10.923525 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:10.923516 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:10.923782 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:10.923498 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:10.923782 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:10.923607 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:10.923782 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:10.923709 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:10.923896 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:10.923780 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:11.007721 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:11.007691 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:11.225268 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:11.225233 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:11.275019 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:11.274983 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:11.416746 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:11.416720 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:11.616442 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:11.616415 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:11.680358 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:11.680336 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:12.223265 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:12.223236 2543 generic.go:356] "Generic (PLEG): container finished" podID="2a0c7205-318e-465c-bd0f-1e4734d2b8e5" containerID="f7fe9b2efb16faabf06deb9ab0a5aa1f8f6e9ea3454b43489843f02e7cbc1388" exitCode=0 Jul 09 00:56:12.223748 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:12.223298 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerDied","Data":"f7fe9b2efb16faabf06deb9ab0a5aa1f8f6e9ea3454b43489843f02e7cbc1388"} Jul 09 00:56:12.573769 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:12.573702 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:12.922886 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:12.922807 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:12.922886 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:12.922819 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:12.923092 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:12.922913 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mpcm" podUID="864bf335-d70a-4510-a93f-cbfb67948f9a" Jul 09 00:56:12.923092 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:12.922956 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69lnl" podUID="c541c940-d9b8-4cb0-ac38-5abdc3e112c8" Jul 09 00:56:12.923092 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:12.923012 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:12.923244 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:12.923118 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thjzt" podUID="b581286c-01e6-45a0-a760-5bb689a8b8e3" Jul 09 00:56:13.227689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:13.227663 2543 generic.go:356] "Generic (PLEG): container finished" podID="2a0c7205-318e-465c-bd0f-1e4734d2b8e5" containerID="fcc5a59d2a3162f69a4241980532bb8c80967e3bfc558d158183a049c4b93bc9" exitCode=0 Jul 09 00:56:13.228232 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:13.227727 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerDied","Data":"fcc5a59d2a3162f69a4241980532bb8c80967e3bfc558d158183a049c4b93bc9"} Jul 09 00:56:13.630827 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:13.630801 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:13.687544 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:13.687516 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:13.948110 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:13.948088 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeReady" Jul 09 00:56:13.948251 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:13.948174 2543 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 00:56:13.990731 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:13.990702 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7h8np"] Jul 09 00:56:14.013118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.013098 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dzg2n"] Jul 09 00:56:14.013229 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.013218 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.015553 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.015530 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 00:56:14.015694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.015568 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 00:56:14.015694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.015603 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7p7kq\"" Jul 09 00:56:14.015694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.015603 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 00:56:14.026870 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.026853 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7h8np"] Jul 09 00:56:14.026950 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.026879 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dzg2n"] Jul 09 00:56:14.026950 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.026901 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h6zd8"] Jul 09 00:56:14.027014 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.026984 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.029262 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.029243 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 00:56:14.029357 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.029315 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95w8r\"" Jul 09 00:56:14.029357 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.029347 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 00:56:14.029443 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.029363 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 00:56:14.029443 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.029367 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 00:56:14.046997 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.046981 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h6zd8"] Jul 09 00:56:14.047076 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.047064 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.049262 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.049248 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 00:56:14.049453 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.049441 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kshp\"" Jul 09 00:56:14.049504 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.049468 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 00:56:14.106340 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.106319 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29569db2-280a-4e93-a06e-0400f7ba321a-cert\") pod \"ingress-canary-7h8np\" (UID: \"29569db2-280a-4e93-a06e-0400f7ba321a\") " pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.106432 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.106356 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmmd\" (UniqueName: \"kubernetes.io/projected/29569db2-280a-4e93-a06e-0400f7ba321a-kube-api-access-zrmmd\") pod \"ingress-canary-7h8np\" (UID: \"29569db2-280a-4e93-a06e-0400f7ba321a\") " pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.119197 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.119175 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:14.169919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.169901 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:14.207204 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207152 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.207204 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207189 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvng\" (UniqueName: \"kubernetes.io/projected/b04a770f-3b16-42aa-8524-42e84978b874-kube-api-access-4gvng\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.207337 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207215 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-crio-socket\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.207337 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207242 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29569db2-280a-4e93-a06e-0400f7ba321a-cert\") pod \"ingress-canary-7h8np\" (UID: \"29569db2-280a-4e93-a06e-0400f7ba321a\") " pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.207337 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207266 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a770f-3b16-42aa-8524-42e84978b874-config-volume\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.207337 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207299 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-data-volume\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.207337 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207325 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04a770f-3b16-42aa-8524-42e84978b874-metrics-tls\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.207522 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207352 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmmd\" (UniqueName: \"kubernetes.io/projected/29569db2-280a-4e93-a06e-0400f7ba321a-kube-api-access-zrmmd\") pod \"ingress-canary-7h8np\" (UID: \"29569db2-280a-4e93-a06e-0400f7ba321a\") " pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.207522 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207382 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a770f-3b16-42aa-8524-42e84978b874-tmp-dir\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.207522 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207431 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.207522 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.207464 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxgxq\" (UniqueName: \"kubernetes.io/projected/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-kube-api-access-xxgxq\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.211262 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.211243 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29569db2-280a-4e93-a06e-0400f7ba321a-cert\") pod \"ingress-canary-7h8np\" (UID: \"29569db2-280a-4e93-a06e-0400f7ba321a\") " pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.215287 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.215262 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmmd\" (UniqueName: \"kubernetes.io/projected/29569db2-280a-4e93-a06e-0400f7ba321a-kube-api-access-zrmmd\") pod \"ingress-canary-7h8np\" (UID: \"29569db2-280a-4e93-a06e-0400f7ba321a\") " pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.232755 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.232735 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" event={"ID":"2a0c7205-318e-465c-bd0f-1e4734d2b8e5","Type":"ContainerStarted","Data":"288ad8ef34007ad4363072a5d0671c75410a8328a7b24b231566da41dfe9c3cd"} Jul 09 00:56:14.253764 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.253729 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wjb6p" podStartSLOduration=9.992785306 podStartE2EDuration="38.253717966s" totalImagesPullingTime="28.26093266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:14.252365374 +0000 UTC m=+38.530204026" watchObservedRunningTime="2026-07-09 00:56:14.253717966 +0000 UTC m=+38.531556618" Jul 09 00:56:14.308541 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308518 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.308612 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308559 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxgxq\" (UniqueName: \"kubernetes.io/projected/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-kube-api-access-xxgxq\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.308698 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308613 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.308698 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308666 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvng\" (UniqueName: \"kubernetes.io/projected/b04a770f-3b16-42aa-8524-42e84978b874-kube-api-access-4gvng\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.308698 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308692 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-crio-socket\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.308854 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308720 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a770f-3b16-42aa-8524-42e84978b874-config-volume\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.308854 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308758 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-data-volume\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.308946 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308911 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-crio-socket\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.308998 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308942 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04a770f-3b16-42aa-8524-42e84978b874-metrics-tls\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.309051 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.308998 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a770f-3b16-42aa-8524-42e84978b874-tmp-dir\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.309098 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.309058 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-data-volume\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.309184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.309161 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.309279 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.309261 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a770f-3b16-42aa-8524-42e84978b874-tmp-dir\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.309331 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.309317 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a770f-3b16-42aa-8524-42e84978b874-config-volume\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.310731 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.310715 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.311001 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.310984 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04a770f-3b16-42aa-8524-42e84978b874-metrics-tls\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.315939 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.315917 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxgxq\" (UniqueName: \"kubernetes.io/projected/56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee-kube-api-access-xxgxq\") pod \"insights-runtime-extractor-dzg2n\" (UID: \"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee\") " pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.316026 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.316010 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvng\" (UniqueName: \"kubernetes.io/projected/b04a770f-3b16-42aa-8524-42e84978b874-kube-api-access-4gvng\") pod \"dns-default-h6zd8\" (UID: \"b04a770f-3b16-42aa-8524-42e84978b874\") " pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.321874 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.321858 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7h8np" Jul 09 00:56:14.335166 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.335152 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dzg2n" Jul 09 00:56:14.354043 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.354019 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:14.376709 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.376689 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:14.478055 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.478018 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:14.522348 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.522322 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dzg2n"] Jul 09 00:56:14.525860 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:14.525811 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b3d4c8_2965_4e45_ba5a_2ae523a4b6ee.slice/crio-9951a68739a99fc952a3a4cfcc84376da1ae257231a03829736faf2cd354ce78 WatchSource:0}: Error finding container 9951a68739a99fc952a3a4cfcc84376da1ae257231a03829736faf2cd354ce78: Status 404 returned error can't find the container with id 9951a68739a99fc952a3a4cfcc84376da1ae257231a03829736faf2cd354ce78 Jul 09 00:56:14.526399 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.526358 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7h8np"] Jul 09 00:56:14.529350 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:14.529292 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29569db2_280a_4e93_a06e_0400f7ba321a.slice/crio-59c83562a3aca34cacf087bb6c2eb50513334c760ef9d77bfc5b1d08648247b9 WatchSource:0}: Error finding container 59c83562a3aca34cacf087bb6c2eb50513334c760ef9d77bfc5b1d08648247b9: Status 404 returned error can't find the container with id 59c83562a3aca34cacf087bb6c2eb50513334c760ef9d77bfc5b1d08648247b9 Jul 09 00:56:14.529877 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.529856 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h6zd8"] Jul 09 00:56:14.532928 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:14.532901 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04a770f_3b16_42aa_8524_42e84978b874.slice/crio-9e3c92cab4f396cf265d537b31eba27c918b1ce2c215d97d94f8806d3aa906f0 WatchSource:0}: Error finding container 9e3c92cab4f396cf265d537b31eba27c918b1ce2c215d97d94f8806d3aa906f0: Status 404 returned error can't find the container with id 9e3c92cab4f396cf265d537b31eba27c918b1ce2c215d97d94f8806d3aa906f0 Jul 09 00:56:14.922635 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.922561 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:14.922784 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.922566 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:14.922784 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.922693 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:14.926597 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.926568 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htn6x\"" Jul 09 00:56:14.926741 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.926727 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 00:56:14.926918 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.926834 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 00:56:14.927118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.927099 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jpd88\"" Jul 09 00:56:14.927173 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.927160 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 00:56:14.927288 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:14.927260 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 00:56:15.239579 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.239445 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h6zd8" event={"ID":"b04a770f-3b16-42aa-8524-42e84978b874","Type":"ContainerStarted","Data":"9e3c92cab4f396cf265d537b31eba27c918b1ce2c215d97d94f8806d3aa906f0"} Jul 09 00:56:15.241922 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.241893 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzg2n" event={"ID":"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee","Type":"ContainerStarted","Data":"9951a68739a99fc952a3a4cfcc84376da1ae257231a03829736faf2cd354ce78"} Jul 09 00:56:15.243612 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.243556 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7h8np" event={"ID":"29569db2-280a-4e93-a06e-0400f7ba321a","Type":"ContainerStarted","Data":"59c83562a3aca34cacf087bb6c2eb50513334c760ef9d77bfc5b1d08648247b9"} Jul 09 00:56:15.252361 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.251456 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-869484c78f-mxxh8"] Jul 09 00:56:15.264464 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.263960 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.264855 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.264823 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869484c78f-mxxh8"] Jul 09 00:56:15.267689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.267452 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 00:56:15.268928 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.268397 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 00:56:15.268928 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.268487 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 00:56:15.268928 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.268676 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 00:56:15.268928 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.268716 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-svrtq\"" Jul 09 00:56:15.268928 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.268740 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 00:56:15.268928 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.268910 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 00:56:15.269275 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.269022 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 00:56:15.416687 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.416628 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-oauth-config\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.416687 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.416692 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-oauth-serving-cert\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.416920 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.416733 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-serving-cert\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.416920 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.416788 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-service-ca\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.416920 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.416845 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-config\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.416920 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.416889 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwv5l\" (UniqueName: \"kubernetes.io/projected/85ee6115-e5eb-4f57-a5d2-310114c98b74-kube-api-access-zwv5l\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.517919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.517418 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-service-ca\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.517919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.517490 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-config\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.517919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.517536 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwv5l\" (UniqueName: \"kubernetes.io/projected/85ee6115-e5eb-4f57-a5d2-310114c98b74-kube-api-access-zwv5l\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.517919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.517887 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-oauth-config\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.518215 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.517927 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-oauth-serving-cert\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.518215 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.517966 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-serving-cert\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.518569 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.518345 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-service-ca\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.519175 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.519050 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-config\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.519175 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.519138 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-oauth-serving-cert\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.523054 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.523015 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-serving-cert\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.524948 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.524922 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-oauth-config\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.526126 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.526080 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwv5l\" (UniqueName: \"kubernetes.io/projected/85ee6115-e5eb-4f57-a5d2-310114c98b74-kube-api-access-zwv5l\") pod \"console-869484c78f-mxxh8\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.577387 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.577357 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:15.664929 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.664898 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:15.714112 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.714082 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:15.773899 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.773824 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869484c78f-mxxh8"] Jul 09 00:56:15.780418 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:15.780385 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ee6115_e5eb_4f57_a5d2_310114c98b74.slice/crio-6e56426a88b9386c67750bd902aca272ade5b915ab530769b565539394459d58 WatchSource:0}: Error finding container 6e56426a88b9386c67750bd902aca272ade5b915ab530769b565539394459d58: Status 404 returned error can't find the container with id 6e56426a88b9386c67750bd902aca272ade5b915ab530769b565539394459d58 Jul 09 00:56:15.967119 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:15.967073 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:16.016888 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.016798 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:16.239524 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.239492 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:16.248301 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.248242 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869484c78f-mxxh8" event={"ID":"85ee6115-e5eb-4f57-a5d2-310114c98b74","Type":"ContainerStarted","Data":"6e56426a88b9386c67750bd902aca272ade5b915ab530769b565539394459d58"} Jul 09 00:56:16.305522 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.305339 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:16.893132 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.893100 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:16.926319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.926297 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c"] Jul 09 00:56:16.949511 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.949489 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c"] Jul 09 00:56:16.949631 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.949577 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:16.951913 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.951894 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mkb9p\"" Jul 09 00:56:16.953673 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.953353 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 00:56:16.953673 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.953406 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 00:56:16.953673 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.953356 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 00:56:16.953902 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.953887 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 00:56:16.953996 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.953971 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 00:56:16.967917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:16.967890 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:17.131949 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.131871 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvnx\" (UniqueName: \"kubernetes.io/projected/2a74b308-8be9-4614-866f-1629491baed0-kube-api-access-pfvnx\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.131949 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.131910 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2a74b308-8be9-4614-866f-1629491baed0-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.131949 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.131936 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a74b308-8be9-4614-866f-1629491baed0-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.132201 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.132094 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a74b308-8be9-4614-866f-1629491baed0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.191686 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.191637 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:17.233054 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.233021 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a74b308-8be9-4614-866f-1629491baed0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.233206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.233109 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvnx\" (UniqueName: \"kubernetes.io/projected/2a74b308-8be9-4614-866f-1629491baed0-kube-api-access-pfvnx\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.233206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.233139 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2a74b308-8be9-4614-866f-1629491baed0-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.233206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.233162 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a74b308-8be9-4614-866f-1629491baed0-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.233894 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.233870 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a74b308-8be9-4614-866f-1629491baed0-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.236687 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.236662 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a74b308-8be9-4614-866f-1629491baed0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.236846 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.236822 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2a74b308-8be9-4614-866f-1629491baed0-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.241179 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.241154 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvnx\" (UniqueName: \"kubernetes.io/projected/2a74b308-8be9-4614-866f-1629491baed0-kube-api-access-pfvnx\") pod \"prometheus-operator-567f7b8b7c-qhd7c\" (UID: \"2a74b308-8be9-4614-866f-1629491baed0\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:17.262037 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:17.262018 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" Jul 09 00:56:18.765226 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:18.765188 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:18.865930 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:18.865897 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:21.922418 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:21.922333 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c"] Jul 09 00:56:21.928958 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:21.928921 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a74b308_8be9_4614_866f_1629491baed0.slice/crio-c7f1efbe66a0b4c82b2e40a72dbc74f8bdcb64c1839823e6e4fafeb4985c824e WatchSource:0}: Error finding container c7f1efbe66a0b4c82b2e40a72dbc74f8bdcb64c1839823e6e4fafeb4985c824e: Status 404 returned error can't find the container with id c7f1efbe66a0b4c82b2e40a72dbc74f8bdcb64c1839823e6e4fafeb4985c824e Jul 09 00:56:22.269778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.269745 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzg2n" event={"ID":"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee","Type":"ContainerStarted","Data":"0e1e4641840b14b2e25728b8d6055804c943b80d61fc52c968a40194338cdf64"} Jul 09 00:56:22.270856 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.270834 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" event={"ID":"2a74b308-8be9-4614-866f-1629491baed0","Type":"ContainerStarted","Data":"c7f1efbe66a0b4c82b2e40a72dbc74f8bdcb64c1839823e6e4fafeb4985c824e"} Jul 09 00:56:22.272159 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.272130 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7h8np" event={"ID":"29569db2-280a-4e93-a06e-0400f7ba321a","Type":"ContainerStarted","Data":"591f67ccc14648ffaa9a71d2997405a03a643b63d89ffb8436b66cbb7e4754b2"} Jul 09 00:56:22.273593 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.273570 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869484c78f-mxxh8" event={"ID":"85ee6115-e5eb-4f57-a5d2-310114c98b74","Type":"ContainerStarted","Data":"4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97"} Jul 09 00:56:22.275209 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.275189 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h6zd8" event={"ID":"b04a770f-3b16-42aa-8524-42e84978b874","Type":"ContainerStarted","Data":"3baeb6eeb05a6abf4301d26da62c4600aa181e1d57b78e36158a20f968837bde"} Jul 09 00:56:22.275295 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.275230 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h6zd8" event={"ID":"b04a770f-3b16-42aa-8524-42e84978b874","Type":"ContainerStarted","Data":"dac672a21b44c327d73cf4349bf1c5ff2c1f091a374a3fa4f3cdfb948c2bf574"} Jul 09 00:56:22.275345 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.275312 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:22.286846 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.286808 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7h8np" podStartSLOduration=2.052072375 podStartE2EDuration="9.286796191s" totalImagesPullingTime="7.234723816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:22.285384093 +0000 UTC m=+46.563222744" watchObservedRunningTime="2026-07-09 00:56:22.286796191 +0000 UTC m=+46.564634841" Jul 09 00:56:22.300445 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.300347 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-869484c78f-mxxh8" podStartSLOduration=1.284614746 podStartE2EDuration="7.300328854s" totalImagesPullingTime="6.015714108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:22.299430879 +0000 UTC m=+46.577269532" watchObservedRunningTime="2026-07-09 00:56:22.300328854 +0000 UTC m=+46.578167508" Jul 09 00:56:22.314674 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.314507 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h6zd8" podStartSLOduration=2.08279632 podStartE2EDuration="9.314491886s" totalImagesPullingTime="7.231695566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:22.313721245 +0000 UTC m=+46.591559895" watchObservedRunningTime="2026-07-09 00:56:22.314491886 +0000 UTC m=+46.592330534" Jul 09 00:56:22.617852 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.617200 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:22.668737 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:22.668681 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:23.281012 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:23.280938 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzg2n" event={"ID":"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee","Type":"ContainerStarted","Data":"2829ab544f7c3ae126e468a1d4dcec0951d8f6ab1ae4dcc440114326db087cb6"} Jul 09 00:56:23.849271 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:23.849187 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:24.126836 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:24.126756 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:24.182425 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:24.182395 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:24.287988 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:24.287942 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" event={"ID":"2a74b308-8be9-4614-866f-1629491baed0","Type":"ContainerStarted","Data":"9050c23a9fae0934d1891f94cc32ccf963887f0587d157924494a0fb1480c271"} Jul 09 00:56:24.288428 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:24.288012 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" event={"ID":"2a74b308-8be9-4614-866f-1629491baed0","Type":"ContainerStarted","Data":"8286c3eb68eee38f06acf3f6ba75da68bd1214fd01d59c18e2f4b37d3e9a9046"} Jul 09 00:56:24.304291 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:24.304242 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qhd7c" podStartSLOduration=6.68874217 podStartE2EDuration="8.304225243s" totalImagesPullingTime="1.615483073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:24.303904836 +0000 UTC m=+48.581743490" watchObservedRunningTime="2026-07-09 00:56:24.304225243 +0000 UTC m=+48.582063895" Jul 09 00:56:25.047132 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:25.047104 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:25.293379 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:25.293311 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzg2n" event={"ID":"56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee","Type":"ContainerStarted","Data":"35673d94a17c8695771a1e21856b6ba63da1d46de1f35e78e3f1690b01c50880"} Jul 09 00:56:25.310362 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:25.310221 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dzg2n" podStartSLOduration=2.025431966 podStartE2EDuration="12.310205573s" totalImagesPullingTime="10.284773607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:56:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:25.309391062 +0000 UTC m=+49.587229711" watchObservedRunningTime="2026-07-09 00:56:25.310205573 +0000 UTC m=+49.588044225" Jul 09 00:56:25.578007 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:25.577938 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:25.578007 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:25.577996 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:25.579042 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:25.579021 2543 patch_prober.go:28] interesting pod/console-869484c78f-mxxh8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 09 00:56:25.579089 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:25.579063 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-869484c78f-mxxh8" podUID="85ee6115-e5eb-4f57-a5d2-310114c98b74" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 09 00:56:26.009305 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.009265 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.236447 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.236418 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.290559 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.290498 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg"] Jul 09 00:56:26.294397 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.294380 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.297575 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.297553 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 00:56:26.297575 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.297567 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 00:56:26.297740 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.297568 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 00:56:26.297740 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.297570 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p84s8\"" Jul 09 00:56:26.302004 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.301989 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-knngl"] Jul 09 00:56:26.305908 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.305889 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.306856 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.306837 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg"] Jul 09 00:56:26.308294 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.308277 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 00:56:26.308500 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.308484 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q82vn\"" Jul 09 00:56:26.308573 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.308490 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 00:56:26.308626 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.308579 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 00:56:26.314553 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.314533 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.380003 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.379984 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.406438 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406418 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.406537 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406462 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-wtmp\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.406537 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406523 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-textfile\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.406690 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406667 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.406739 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406714 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-accelerators-collector-config\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.406778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406743 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.406778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406763 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzx4j\" (UniqueName: \"kubernetes.io/projected/7b21af0f-9034-4a91-b7c2-27b34cce3685-kube-api-access-tzx4j\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.406838 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406819 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb525-bbe6-4c26-b8eb-cf1505989264-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.406868 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406845 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b21af0f-9034-4a91-b7c2-27b34cce3685-metrics-client-ca\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.406901 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406867 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmq4r\" (UniqueName: \"kubernetes.io/projected/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-api-access-hmq4r\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.406933 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406907 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6ceb525-bbe6-4c26-b8eb-cf1505989264-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.406965 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406951 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-tls\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.407009 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.406988 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.407059 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.407011 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-sys\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.407059 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.407029 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-root\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.478971 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.478939 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.507881 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.507856 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb525-bbe6-4c26-b8eb-cf1505989264-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.507976 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.507891 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b21af0f-9034-4a91-b7c2-27b34cce3685-metrics-client-ca\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.507976 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.507922 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmq4r\" (UniqueName: \"kubernetes.io/projected/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-api-access-hmq4r\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.507976 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.507950 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6ceb525-bbe6-4c26-b8eb-cf1505989264-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.508131 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508000 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-tls\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508131 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508056 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.508131 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508086 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-sys\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508131 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508113 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-root\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508140 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508171 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-wtmp\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:26.508179 2543 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508185 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-root\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508220 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-textfile\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:26.508250 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-tls podName:7b21af0f-9034-4a91-b7c2-27b34cce3685 nodeName:}" failed. No retries permitted until 2026-07-09 00:56:27.008230324 +0000 UTC m=+51.286068967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-tls") pod "node-exporter-knngl" (UID: "7b21af0f-9034-4a91-b7c2-27b34cce3685") : secret "node-exporter-tls" not found Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:26.508284 2543 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508289 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb525-bbe6-4c26-b8eb-cf1505989264-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508301 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508333 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-wtmp\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:26.508342 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-tls podName:a6ceb525-bbe6-4c26-b8eb-cf1505989264 nodeName:}" failed. No retries permitted until 2026-07-09 00:56:27.008325461 +0000 UTC m=+51.286164158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-tzsdg" (UID: "a6ceb525-bbe6-4c26-b8eb-cf1505989264") : secret "kube-state-metrics-tls" not found Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508378 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-accelerators-collector-config\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.508694 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508409 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.509184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508415 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b21af0f-9034-4a91-b7c2-27b34cce3685-sys\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.509184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508442 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzx4j\" (UniqueName: \"kubernetes.io/projected/7b21af0f-9034-4a91-b7c2-27b34cce3685-kube-api-access-tzx4j\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.509184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508502 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-textfile\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.509184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508583 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b21af0f-9034-4a91-b7c2-27b34cce3685-metrics-client-ca\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.509184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508937 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-accelerators-collector-config\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.509184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.508997 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6ceb525-bbe6-4c26-b8eb-cf1505989264-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.509184 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.509042 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.510625 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.510607 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.510710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.510636 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.517955 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.517936 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzx4j\" (UniqueName: \"kubernetes.io/projected/7b21af0f-9034-4a91-b7c2-27b34cce3685-kube-api-access-tzx4j\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:26.518441 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.518421 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmq4r\" (UniqueName: \"kubernetes.io/projected/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-api-access-hmq4r\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:26.579768 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.579712 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.701313 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.701288 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.802411 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.802386 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:26.907662 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:26.904388 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:27.011208 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.011185 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-tls\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:27.011338 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.011224 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:27.013460 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.013431 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b21af0f-9034-4a91-b7c2-27b34cce3685-node-exporter-tls\") pod \"node-exporter-knngl\" (UID: \"7b21af0f-9034-4a91-b7c2-27b34cce3685\") " pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:27.013572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.013556 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb525-bbe6-4c26-b8eb-cf1505989264-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tzsdg\" (UID: \"a6ceb525-bbe6-4c26-b8eb-cf1505989264\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:27.201217 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.201188 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:27.204059 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.204035 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" Jul 09 00:56:27.215273 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.215250 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-knngl" Jul 09 00:56:27.226147 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:27.226120 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b21af0f_9034_4a91_b7c2_27b34cce3685.slice/crio-8e98808a3ac733c807567cfa640a289085018e3670753dabfcaba860cb556774 WatchSource:0}: Error finding container 8e98808a3ac733c807567cfa640a289085018e3670753dabfcaba860cb556774: Status 404 returned error can't find the container with id 8e98808a3ac733c807567cfa640a289085018e3670753dabfcaba860cb556774 Jul 09 00:56:27.302714 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.302420 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-knngl" event={"ID":"7b21af0f-9034-4a91-b7c2-27b34cce3685","Type":"ContainerStarted","Data":"8e98808a3ac733c807567cfa640a289085018e3670753dabfcaba860cb556774"} Jul 09 00:56:27.317800 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.317775 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 00:56:27.322135 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.322119 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.324600 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324577 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z6wbb\"" Jul 09 00:56:27.324600 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324592 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 00:56:27.324792 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324615 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 00:56:27.324792 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324613 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 00:56:27.324895 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324870 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 00:56:27.324951 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324899 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 00:56:27.324951 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324919 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 00:56:27.324951 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324901 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 00:56:27.325084 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.324904 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 00:56:27.325296 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.325278 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 00:56:27.335105 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.335084 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 00:56:27.336231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.336212 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg"] Jul 09 00:56:27.338067 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:27.338047 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ceb525_bbe6_4c26_b8eb_cf1505989264.slice/crio-c2117e7308f58ea226dab072e50edbb93815e5da4b037a82bc572cf6cf10c906 WatchSource:0}: Error finding container c2117e7308f58ea226dab072e50edbb93815e5da4b037a82bc572cf6cf10c906: Status 404 returned error can't find the container with id c2117e7308f58ea226dab072e50edbb93815e5da4b037a82bc572cf6cf10c906 Jul 09 00:56:27.498210 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.498138 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:27.514428 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514405 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514560 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514434 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514560 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514452 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514560 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514493 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514672 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514584 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-web-config\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514672 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514620 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514672 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514653 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514764 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514679 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514796 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514760 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514796 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514781 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-out\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514855 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514802 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514855 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514818 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkt4g\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-kube-api-access-hkt4g\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.514855 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.514846 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.553007 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.552982 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:27.615809 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.615788 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.615942 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.615823 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.615942 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.615865 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.615942 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.615901 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-web-config\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.615942 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.615933 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616148 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.615959 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616148 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.615987 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616148 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.616057 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616148 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.616090 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-out\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616148 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.616127 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616376 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.616150 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkt4g\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-kube-api-access-hkt4g\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616376 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.616222 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616376 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.616251 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.616933 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.616769 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.617859 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.617201 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.617859 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.617556 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.619033 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.619009 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.619132 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.619009 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-web-config\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.619189 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.619125 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.619250 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.619233 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.619412 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.619387 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.619754 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.619739 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.620206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.620187 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.620326 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.620311 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-out\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.620765 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.620749 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.624338 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.624322 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkt4g\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-kube-api-access-hkt4g\") pod \"alertmanager-main-0\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.631206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.631188 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:56:27.757861 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:27.757794 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 00:56:27.761068 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:27.761042 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f71c45_83e0_40f4_9032_a68e5a91d666.slice/crio-d06e807296333af799558250cfdeb00565592ad3b359005e8820e6f6b8a30ad6 WatchSource:0}: Error finding container d06e807296333af799558250cfdeb00565592ad3b359005e8820e6f6b8a30ad6: Status 404 returned error can't find the container with id d06e807296333af799558250cfdeb00565592ad3b359005e8820e6f6b8a30ad6 Jul 09 00:56:28.308063 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:28.308026 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerStarted","Data":"d06e807296333af799558250cfdeb00565592ad3b359005e8820e6f6b8a30ad6"} Jul 09 00:56:28.309602 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:28.309570 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" event={"ID":"a6ceb525-bbe6-4c26-b8eb-cf1505989264","Type":"ContainerStarted","Data":"c2117e7308f58ea226dab072e50edbb93815e5da4b037a82bc572cf6cf10c906"} Jul 09 00:56:28.601458 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:28.601368 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:28.702416 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:28.702370 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:29.037325 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.037288 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:29.140359 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.140331 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:29.238018 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.237993 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:29.242976 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.242955 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bf8455844-bsdkw"] Jul 09 00:56:29.246178 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.246159 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.248538 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.248521 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8na312saqvler\"" Jul 09 00:56:29.248670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.248525 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r4j77\"" Jul 09 00:56:29.248670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.248607 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 09 00:56:29.250882 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.250861 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 09 00:56:29.251124 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.251110 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 09 00:56:29.252614 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.252562 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 09 00:56:29.253518 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.253500 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 09 00:56:29.276972 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.276948 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bf8455844-bsdkw"] Jul 09 00:56:29.331081 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331055 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.331436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331094 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f84cb37f-3c84-45fe-bca4-61b562dcec18-metrics-client-ca\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.331436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331131 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.331436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331189 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-grpc-tls\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.331436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331231 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4gm5\" (UniqueName: \"kubernetes.io/projected/f84cb37f-3c84-45fe-bca4-61b562dcec18-kube-api-access-h4gm5\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.331436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331323 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.331921 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331475 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-tls\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.331921 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.331510 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.432441 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.432411 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-grpc-tls\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.432569 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.432448 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gm5\" (UniqueName: \"kubernetes.io/projected/f84cb37f-3c84-45fe-bca4-61b562dcec18-kube-api-access-h4gm5\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.432569 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.432500 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.432727 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.432567 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-tls\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.432727 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.432592 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.432727 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.432633 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.432727 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.432674 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f84cb37f-3c84-45fe-bca4-61b562dcec18-metrics-client-ca\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.433241 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.433153 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.433457 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.433423 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f84cb37f-3c84-45fe-bca4-61b562dcec18-metrics-client-ca\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.435778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.435571 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-grpc-tls\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.436060 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.436034 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.436167 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.436143 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-tls\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.436236 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.436171 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.436292 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.436234 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.436343 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.436299 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f84cb37f-3c84-45fe-bca4-61b562dcec18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.439999 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.439976 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gm5\" (UniqueName: \"kubernetes.io/projected/f84cb37f-3c84-45fe-bca4-61b562dcec18-kube-api-access-h4gm5\") pod \"thanos-querier-bf8455844-bsdkw\" (UID: \"f84cb37f-3c84-45fe-bca4-61b562dcec18\") " pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.466411 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.466375 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:29.556928 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.556904 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:29.684756 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.684734 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bf8455844-bsdkw"] Jul 09 00:56:29.687505 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:29.687464 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84cb37f_3c84_45fe_bca4_61b562dcec18.slice/crio-f7769e6fd4b9c0b338921eb0e4c052ed390c1d2861957ad77af701195c47992a WatchSource:0}: Error finding container f7769e6fd4b9c0b338921eb0e4c052ed390c1d2861957ad77af701195c47992a: Status 404 returned error can't find the container with id f7769e6fd4b9c0b338921eb0e4c052ed390c1d2861957ad77af701195c47992a Jul 09 00:56:29.838732 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:29.838657 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:30.317811 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.317774 2543 generic.go:356] "Generic (PLEG): container finished" podID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerID="cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85" exitCode=0 Jul 09 00:56:30.317984 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.317869 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85"} Jul 09 00:56:30.319592 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.319570 2543 generic.go:356] "Generic (PLEG): container finished" podID="7b21af0f-9034-4a91-b7c2-27b34cce3685" containerID="2a596510fa61fe8ed61caa6609b7d0003dbf466cd9d6f2ca9e3b8cce658c8c95" exitCode=0 Jul 09 00:56:30.319704 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.319671 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-knngl" event={"ID":"7b21af0f-9034-4a91-b7c2-27b34cce3685","Type":"ContainerDied","Data":"2a596510fa61fe8ed61caa6609b7d0003dbf466cd9d6f2ca9e3b8cce658c8c95"} Jul 09 00:56:30.322943 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.322741 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" event={"ID":"a6ceb525-bbe6-4c26-b8eb-cf1505989264","Type":"ContainerStarted","Data":"3bfe700913f77591e2aaff869d2419ddf685e77608f9cd9ae9b9a135c4a7e38e"} Jul 09 00:56:30.322943 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.322792 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" event={"ID":"a6ceb525-bbe6-4c26-b8eb-cf1505989264","Type":"ContainerStarted","Data":"962316ea839e6ee5864b5cae92b21f142531c91bfab5c2ed6ae2b8dba4e8811c"} Jul 09 00:56:30.322943 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.322834 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" event={"ID":"a6ceb525-bbe6-4c26-b8eb-cf1505989264","Type":"ContainerStarted","Data":"fa988c2e20044514072131995c2a036ead9113c7fa9bbc29baa9f1ff086041a9"} Jul 09 00:56:30.324307 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.324286 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" event={"ID":"f84cb37f-3c84-45fe-bca4-61b562dcec18","Type":"ContainerStarted","Data":"f7769e6fd4b9c0b338921eb0e4c052ed390c1d2861957ad77af701195c47992a"} Jul 09 00:56:30.384841 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:30.384791 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tzsdg" podStartSLOduration=2.483075994 podStartE2EDuration="4.384773427s" totalImagesPullingTime="1.901697433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:30.363978506 +0000 UTC m=+54.641817158" watchObservedRunningTime="2026-07-09 00:56:30.384773427 +0000 UTC m=+54.662612078" Jul 09 00:56:31.023993 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.023962 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:31.033457 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.033242 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw"] Jul 09 00:56:31.036590 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.036569 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.039342 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.039245 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 00:56:31.039736 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.039468 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lx6hb\"" Jul 09 00:56:31.043762 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.043734 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw"] Jul 09 00:56:31.046215 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.045979 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf817aca-fa10-4b4f-967c-979c65d5bd64-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-w27gw\" (UID: \"cf817aca-fa10-4b4f-967c-979c65d5bd64\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.046215 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.046029 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6thfd\" (UniqueName: \"kubernetes.io/projected/cf817aca-fa10-4b4f-967c-979c65d5bd64-kube-api-access-6thfd\") pod \"monitoring-plugin-d5dcff7cf-w27gw\" (UID: \"cf817aca-fa10-4b4f-967c-979c65d5bd64\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.123171 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.123127 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:31.146594 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.146571 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf817aca-fa10-4b4f-967c-979c65d5bd64-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-w27gw\" (UID: \"cf817aca-fa10-4b4f-967c-979c65d5bd64\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.146736 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.146611 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6thfd\" (UniqueName: \"kubernetes.io/projected/cf817aca-fa10-4b4f-967c-979c65d5bd64-kube-api-access-6thfd\") pod \"monitoring-plugin-d5dcff7cf-w27gw\" (UID: \"cf817aca-fa10-4b4f-967c-979c65d5bd64\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.149179 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.149152 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf817aca-fa10-4b4f-967c-979c65d5bd64-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-w27gw\" (UID: \"cf817aca-fa10-4b4f-967c-979c65d5bd64\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.154656 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.154622 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6thfd\" (UniqueName: \"kubernetes.io/projected/cf817aca-fa10-4b4f-967c-979c65d5bd64-kube-api-access-6thfd\") pod \"monitoring-plugin-d5dcff7cf-w27gw\" (UID: \"cf817aca-fa10-4b4f-967c-979c65d5bd64\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.177089 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.177066 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:31.331287 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.331224 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-knngl" event={"ID":"7b21af0f-9034-4a91-b7c2-27b34cce3685","Type":"ContainerStarted","Data":"b47b0d6d710b88d75268ff4f497a255d73b5884f115f9496263038fc3c0752ea"} Jul 09 00:56:31.331392 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.331286 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-knngl" event={"ID":"7b21af0f-9034-4a91-b7c2-27b34cce3685","Type":"ContainerStarted","Data":"eec319da974e3d44cbb3f758dc6cda794fe1b4a0743835e472f404c37c5b3cbb"} Jul 09 00:56:31.350927 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.350888 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:31.351069 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.351023 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-knngl" podStartSLOduration=3.265899728 podStartE2EDuration="5.351006997s" totalImagesPullingTime="2.013749895s" totalInitContainerRuntime="71.357374ms" isStatefulPod=true podCreationTimestamp="2026-07-09 00:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:31.350085924 +0000 UTC m=+55.627924577" watchObservedRunningTime="2026-07-09 00:56:31.351006997 +0000 UTC m=+55.628845646" Jul 09 00:56:31.492041 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.491993 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw"] Jul 09 00:56:31.765151 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.765112 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:31.866495 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:31.866453 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:32.072455 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:32.072389 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf817aca_fa10_4b4f_967c_979c65d5bd64.slice/crio-29598d64579a76206a0233e6d1c31afb7b06f24e186f9afdb56da2a840225185 WatchSource:0}: Error finding container 29598d64579a76206a0233e6d1c31afb7b06f24e186f9afdb56da2a840225185: Status 404 returned error can't find the container with id 29598d64579a76206a0233e6d1c31afb7b06f24e186f9afdb56da2a840225185 Jul 09 00:56:32.105343 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.105301 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:32.162098 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.162069 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:32.285618 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.284214 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h6zd8" Jul 09 00:56:32.340489 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.340376 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" event={"ID":"cf817aca-fa10-4b4f-967c-979c65d5bd64","Type":"ContainerStarted","Data":"29598d64579a76206a0233e6d1c31afb7b06f24e186f9afdb56da2a840225185"} Jul 09 00:56:32.435207 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.435066 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:32.438353 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.438325 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:56:32.466699 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.466548 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:56:32.466699 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.466636 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.469517 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.469284 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 00:56:32.469517 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.469374 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-41ltc3jf8fgj0\"" Jul 09 00:56:32.469754 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.469596 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 00:56:32.469754 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.469675 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2hz55\"" Jul 09 00:56:32.470393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.469924 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 00:56:32.470393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.469962 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 00:56:32.470393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.470043 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 00:56:32.470393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.470063 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 00:56:32.470393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.470214 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 00:56:32.470393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.470227 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 00:56:32.470393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.470264 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 00:56:32.470832 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.470403 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 00:56:32.470883 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.470874 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 00:56:32.471250 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.471043 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 00:56:32.476829 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.474930 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 00:56:32.486872 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.485423 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 00:56:32.486872 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.486502 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 00:56:32.503130 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.503103 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:32.560658 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560621 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.560804 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560665 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.560804 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560690 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.560804 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560748 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.560917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560850 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.560917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560877 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.560917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560905 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561028 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.560967 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561028 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561007 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561028 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561026 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-config\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561182 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561068 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561182 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561097 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561182 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561120 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561182 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561141 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487n4\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-kube-api-access-487n4\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561182 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561171 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561223 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561245 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561264 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561280 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.561436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.561318 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662136 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662054 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662136 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662099 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662136 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662124 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662394 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662155 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662394 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662181 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662686 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662625 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662827 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662710 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662827 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662743 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662827 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662783 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.662827 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662823 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663053 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662888 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663053 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662926 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663053 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662951 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663053 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.662982 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663053 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663004 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663297 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663068 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-config\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663297 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663095 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663297 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663130 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663297 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663174 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663297 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663223 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487n4\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-kube-api-access-487n4\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.663297 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663291 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.663947 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.664614 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.665187 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.665473 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.665753 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.666410 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.666595 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.666805 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667052 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.666909 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.667259 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.667572 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.667321 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.668130 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.668113 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.669766 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.669564 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.670054 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.670030 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.670448 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.670402 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.670771 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.670708 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.671006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.670968 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.671704 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.671683 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487n4\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-kube-api-access-487n4\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.671814 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.671792 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-config\") pod \"prometheus-k8s-0\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:32.790103 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:32.790065 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:33.037192 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.037098 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:56:33.038397 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:33.038374 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e41e0a6_e9b0_4163_ad61_1586ab460586.slice/crio-bf3f9061291e78a4a8ccbb326331f1ed9a98bb4a93be4be19162d23e85cf1e88 WatchSource:0}: Error finding container bf3f9061291e78a4a8ccbb326331f1ed9a98bb4a93be4be19162d23e85cf1e88: Status 404 returned error can't find the container with id bf3f9061291e78a4a8ccbb326331f1ed9a98bb4a93be4be19162d23e85cf1e88 Jul 09 00:56:33.347229 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.347201 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" event={"ID":"f84cb37f-3c84-45fe-bca4-61b562dcec18","Type":"ContainerStarted","Data":"5afa96d1d2dcf12a1bd71965f6e238f5053669d663f2da05d259f142bbd2a3f5"} Jul 09 00:56:33.347348 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.347281 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" event={"ID":"f84cb37f-3c84-45fe-bca4-61b562dcec18","Type":"ContainerStarted","Data":"60105fcfef1b4414f0cd1fa68e032a7ad66099677a43e572023de0b72f40529a"} Jul 09 00:56:33.347348 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.347318 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" event={"ID":"f84cb37f-3c84-45fe-bca4-61b562dcec18","Type":"ContainerStarted","Data":"2f8c4b3d3525568e36f13b416701f76dc0c4e8ef55b4b2072eb3e168cc5c1700"} Jul 09 00:56:33.350046 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.350017 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerStarted","Data":"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250"} Jul 09 00:56:33.350163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.350080 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerStarted","Data":"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c"} Jul 09 00:56:33.350163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.350111 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerStarted","Data":"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496"} Jul 09 00:56:33.351785 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.351759 2543 generic.go:356] "Generic (PLEG): container finished" podID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerID="186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0" exitCode=0 Jul 09 00:56:33.351897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.351811 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0"} Jul 09 00:56:33.351897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.351867 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerStarted","Data":"bf3f9061291e78a4a8ccbb326331f1ed9a98bb4a93be4be19162d23e85cf1e88"} Jul 09 00:56:33.440456 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.440430 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869484c78f-mxxh8"] Jul 09 00:56:33.639172 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.639138 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:33.740684 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:33.740535 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:34.358808 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:34.358726 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" event={"ID":"cf817aca-fa10-4b4f-967c-979c65d5bd64","Type":"ContainerStarted","Data":"c2140b5e888f12c0a96765ab94c1d8a9bd303e6a84e79bf52401533dedd047f3"} Jul 09 00:56:34.359115 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:34.359099 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:34.366691 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:34.366659 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerStarted","Data":"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed"} Jul 09 00:56:34.366834 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:34.366737 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerStarted","Data":"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b"} Jul 09 00:56:34.367691 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:34.367670 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" Jul 09 00:56:34.375796 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:34.375746 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-w27gw" podStartSLOduration=1.4321015209999999 podStartE2EDuration="3.375727675s" totalImagesPullingTime="1.943626154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:34.373807885 +0000 UTC m=+58.651646542" watchObservedRunningTime="2026-07-09 00:56:34.375727675 +0000 UTC m=+58.653566327" Jul 09 00:56:35.375901 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.375803 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerStarted","Data":"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd"} Jul 09 00:56:35.379305 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.379282 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" event={"ID":"f84cb37f-3c84-45fe-bca4-61b562dcec18","Type":"ContainerStarted","Data":"75b1d45dced4c1eddae41387325264d9c2d79bcd14af66b13e0a0684ac7578b2"} Jul 09 00:56:35.379424 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.379341 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" event={"ID":"f84cb37f-3c84-45fe-bca4-61b562dcec18","Type":"ContainerStarted","Data":"e7906446b4d6441aee42b427706e0ee0f3dc8472fb16ab8ef27feb6ddfc3b098"} Jul 09 00:56:35.379424 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.379384 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" event={"ID":"f84cb37f-3c84-45fe-bca4-61b562dcec18","Type":"ContainerStarted","Data":"4f55b8fc22307d1143c3f7ff276ba2b231d30d7bc54a777716c43a325004f0c5"} Jul 09 00:56:35.379548 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.379520 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:35.407770 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.407570 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.070723515 podStartE2EDuration="8.407550514s" totalImagesPullingTime="5.31553715s" totalInitContainerRuntime="21.289849ms" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:35.40741088 +0000 UTC m=+59.685249533" watchObservedRunningTime="2026-07-09 00:56:35.407550514 +0000 UTC m=+59.685389166" Jul 09 00:56:35.437659 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.437579 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" podStartSLOduration=1.7830837000000002 podStartE2EDuration="6.437558453s" totalImagesPullingTime="4.654474753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:35.435014887 +0000 UTC m=+59.712853550" watchObservedRunningTime="2026-07-09 00:56:35.437558453 +0000 UTC m=+59.715397107" Jul 09 00:56:35.739165 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.739119 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:35.839286 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:35.839253 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:36.990654 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:36.990620 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:37.091220 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:37.091187 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:37.393702 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:37.393677 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerStarted","Data":"0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694"} Jul 09 00:56:37.393808 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:37.393789 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerStarted","Data":"9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb"} Jul 09 00:56:37.394002 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:37.393841 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerStarted","Data":"dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a"} Jul 09 00:56:37.394002 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:37.393879 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerStarted","Data":"f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed"} Jul 09 00:56:38.400517 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:38.400482 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerStarted","Data":"b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223"} Jul 09 00:56:38.400984 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:38.400538 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerStarted","Data":"a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9"} Jul 09 00:56:38.432041 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:38.431989 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.725691249 podStartE2EDuration="6.431970521s" totalImagesPullingTime="3.691666866s" totalInitContainerRuntime="14.612406ms" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:38.429959213 +0000 UTC m=+62.707797863" watchObservedRunningTime="2026-07-09 00:56:38.431970521 +0000 UTC m=+62.709809173" Jul 09 00:56:39.225700 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:39.225672 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqlk7" Jul 09 00:56:39.346009 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:39.345983 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:39.436294 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:39.436262 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:40.827262 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:40.827232 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:40.927812 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:40.927784 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:41.043141 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.043116 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:41.043277 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.043170 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:41.045589 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.045572 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 00:56:41.045667 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.045601 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 00:56:41.055718 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.055697 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c541c940-d9b8-4cb0-ac38-5abdc3e112c8-metrics-certs\") pod \"network-metrics-daemon-69lnl\" (UID: \"c541c940-d9b8-4cb0-ac38-5abdc3e112c8\") " pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:41.055816 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.055738 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b581286c-01e6-45a0-a760-5bb689a8b8e3-original-pull-secret\") pod \"global-pull-secret-syncer-thjzt\" (UID: \"b581286c-01e6-45a0-a760-5bb689a8b8e3\") " pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:41.143865 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.143787 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:41.145971 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.145952 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 00:56:41.156749 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.156728 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 00:56:41.167730 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.167707 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrhbb\" (UniqueName: \"kubernetes.io/projected/864bf335-d70a-4510-a93f-cbfb67948f9a-kube-api-access-zrhbb\") pod \"network-check-target-2mpcm\" (UID: \"864bf335-d70a-4510-a93f-cbfb67948f9a\") " pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:41.336729 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.336702 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htn6x\"" Jul 09 00:56:41.342472 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.342452 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jpd88\"" Jul 09 00:56:41.345224 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.345199 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thjzt" Jul 09 00:56:41.345299 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.345224 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:41.350987 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.350968 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69lnl" Jul 09 00:56:41.394210 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.394102 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bf8455844-bsdkw" Jul 09 00:56:41.528272 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.528157 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2mpcm"] Jul 09 00:56:41.531528 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:41.531496 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864bf335_d70a_4510_a93f_cbfb67948f9a.slice/crio-b45aab85785a85cc96ac27ea1ae0b4fc7cea00a6affcdf6c9e4cc9de2bdb05bf WatchSource:0}: Error finding container b45aab85785a85cc96ac27ea1ae0b4fc7cea00a6affcdf6c9e4cc9de2bdb05bf: Status 404 returned error can't find the container with id b45aab85785a85cc96ac27ea1ae0b4fc7cea00a6affcdf6c9e4cc9de2bdb05bf Jul 09 00:56:41.750099 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.750073 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-thjzt"] Jul 09 00:56:41.752404 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:41.752369 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-69lnl"] Jul 09 00:56:41.752763 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:41.752743 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb581286c_01e6_45a0_a760_5bb689a8b8e3.slice/crio-4bb125870eadfb8a889265df6bd1cfdf9b2d78515ace682d4822ef0a1ae55245 WatchSource:0}: Error finding container 4bb125870eadfb8a889265df6bd1cfdf9b2d78515ace682d4822ef0a1ae55245: Status 404 returned error can't find the container with id 4bb125870eadfb8a889265df6bd1cfdf9b2d78515ace682d4822ef0a1ae55245 Jul 09 00:56:41.755007 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:56:41.754977 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc541c940_d9b8_4cb0_ac38_5abdc3e112c8.slice/crio-787662fb6df0a790c218ed6adea162b09900ef7aabe976d816e3ac275fe693ca WatchSource:0}: Error finding container 787662fb6df0a790c218ed6adea162b09900ef7aabe976d816e3ac275fe693ca: Status 404 returned error can't find the container with id 787662fb6df0a790c218ed6adea162b09900ef7aabe976d816e3ac275fe693ca Jul 09 00:56:42.424571 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:42.424494 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-thjzt" event={"ID":"b581286c-01e6-45a0-a760-5bb689a8b8e3","Type":"ContainerStarted","Data":"4bb125870eadfb8a889265df6bd1cfdf9b2d78515ace682d4822ef0a1ae55245"} Jul 09 00:56:42.426633 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:42.426548 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69lnl" event={"ID":"c541c940-d9b8-4cb0-ac38-5abdc3e112c8","Type":"ContainerStarted","Data":"787662fb6df0a790c218ed6adea162b09900ef7aabe976d816e3ac275fe693ca"} Jul 09 00:56:42.429908 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:42.429829 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2mpcm" event={"ID":"864bf335-d70a-4510-a93f-cbfb67948f9a","Type":"ContainerStarted","Data":"b45aab85785a85cc96ac27ea1ae0b4fc7cea00a6affcdf6c9e4cc9de2bdb05bf"} Jul 09 00:56:42.791196 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:42.790225 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:56:43.443065 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.442948 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69lnl" event={"ID":"c541c940-d9b8-4cb0-ac38-5abdc3e112c8","Type":"ContainerStarted","Data":"fbf71d6bcad622360db3baf5ab6997c44b1e868ee0de847c4ab4e99e696cb8ed"} Jul 09 00:56:43.913384 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.912836 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8578cb8667-gxccm"] Jul 09 00:56:43.915485 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.915453 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:43.931773 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.931743 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8578cb8667-gxccm"] Jul 09 00:56:43.932671 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.932412 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 00:56:43.973890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.972465 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-oauth-config\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:43.973890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.972528 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-serving-cert\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:43.973890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.972613 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-oauth-serving-cert\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:43.973890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.972675 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-config\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:43.973890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.972734 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmhj\" (UniqueName: \"kubernetes.io/projected/326496df-f7b7-4eff-a9c4-b8294cbeaca6-kube-api-access-lzmhj\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:43.973890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.972802 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-service-ca\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:43.973890 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:43.972833 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-trusted-ca-bundle\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.074094 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074054 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-service-ca\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.074293 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074109 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-trusted-ca-bundle\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.074293 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074156 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-oauth-config\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.074293 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074195 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-serving-cert\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.074293 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074229 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-oauth-serving-cert\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.074293 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074261 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-config\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.074568 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074300 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmhj\" (UniqueName: \"kubernetes.io/projected/326496df-f7b7-4eff-a9c4-b8294cbeaca6-kube-api-access-lzmhj\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.075057 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.074944 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-service-ca\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.075202 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.075180 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-trusted-ca-bundle\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.075824 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.075709 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-oauth-serving-cert\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.075933 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.075827 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-config\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.077911 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.077878 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-oauth-config\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.078075 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.077887 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-serving-cert\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.085289 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.085267 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmhj\" (UniqueName: \"kubernetes.io/projected/326496df-f7b7-4eff-a9c4-b8294cbeaca6-kube-api-access-lzmhj\") pod \"console-8578cb8667-gxccm\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.235876 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.235844 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:44.262298 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.262227 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:44.382339 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.382235 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8578cb8667-gxccm"] Jul 09 00:56:44.450199 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.450007 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69lnl" event={"ID":"c541c940-d9b8-4cb0-ac38-5abdc3e112c8","Type":"ContainerStarted","Data":"bc227d7ae89ac4ad0aa7242c641fea94d37c51f50f6937f40afec3acfd2c6a96"} Jul 09 00:56:44.451414 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.451384 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578cb8667-gxccm" event={"ID":"326496df-f7b7-4eff-a9c4-b8294cbeaca6","Type":"ContainerStarted","Data":"8d1424fe50c6f6f27609a701fba99e397e25d92a1a0a8a10eade14b35c38a3a9"} Jul 09 00:56:44.466486 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.466413 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-69lnl" podStartSLOduration=66.971757592 podStartE2EDuration="1m8.466392345s" totalImagesPullingTime="1.494634753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:44.465431304 +0000 UTC m=+68.743269955" watchObservedRunningTime="2026-07-09 00:56:44.466392345 +0000 UTC m=+68.744230977" Jul 09 00:56:44.551130 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.550354 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:44.648806 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.648776 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:44.863127 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.863044 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:44.914661 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:44.914609 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:45.458665 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:45.458621 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578cb8667-gxccm" event={"ID":"326496df-f7b7-4eff-a9c4-b8294cbeaca6","Type":"ContainerStarted","Data":"995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd"} Jul 09 00:56:45.478378 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:45.478334 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8578cb8667-gxccm" podStartSLOduration=2.478317288 podStartE2EDuration="2.478317288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:45.477162215 +0000 UTC m=+69.755000903" watchObservedRunningTime="2026-07-09 00:56:45.478317288 +0000 UTC m=+69.756155939" Jul 09 00:56:45.677681 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:45.677628 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:46.648674 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:46.648620 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:46.697946 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:46.697916 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:47.469788 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:47.469750 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-thjzt" event={"ID":"b581286c-01e6-45a0-a760-5bb689a8b8e3","Type":"ContainerStarted","Data":"a3a5f92ee00c8c2fc7120bbdfb6847b100aa1c2c8f5afdd18b1a265b2f695f21"} Jul 09 00:56:47.471330 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:47.471304 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2mpcm" event={"ID":"864bf335-d70a-4510-a93f-cbfb67948f9a","Type":"ContainerStarted","Data":"093f94b085e02c09b04d2c07c97b881f246241c51e5c9a019556b89263a17222"} Jul 09 00:56:47.471449 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:47.471432 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:56:47.484222 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:47.484178 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-thjzt" podStartSLOduration=65.355019248 podStartE2EDuration="1m10.4841643s" totalImagesPullingTime="5.129145052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 00:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:47.483253276 +0000 UTC m=+71.761091928" watchObservedRunningTime="2026-07-09 00:56:47.4841643 +0000 UTC m=+71.762002951" Jul 09 00:56:47.497711 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:47.497664 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2mpcm" podStartSLOduration=66.152281986 podStartE2EDuration="1m11.497635518s" totalImagesPullingTime="5.345353532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:56:47.495929097 +0000 UTC m=+71.773767749" watchObservedRunningTime="2026-07-09 00:56:47.497635518 +0000 UTC m=+71.775474166" Jul 09 00:56:49.857915 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:49.857882 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:49.906197 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:49.906164 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:50.716833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:50.716804 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:50.768715 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:50.768687 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:54.236243 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:54.236210 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:54.236605 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:54.236299 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:54.240904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:54.240886 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:54.500899 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:54.500829 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:56:54.642661 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:54.642616 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:54.709436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:54.709407 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:54.759307 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:54.759222 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:56:58.479237 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.478824 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-869484c78f-mxxh8" podUID="85ee6115-e5eb-4f57-a5d2-310114c98b74" containerName="console" containerID="cri-o://4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97" gracePeriod=15 Jul 09 00:56:58.722660 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.722627 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:58.800102 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800039 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-serving-cert\") pod \"85ee6115-e5eb-4f57-a5d2-310114c98b74\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " Jul 09 00:56:58.800215 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800102 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-service-ca\") pod \"85ee6115-e5eb-4f57-a5d2-310114c98b74\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " Jul 09 00:56:58.800215 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800145 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-config\") pod \"85ee6115-e5eb-4f57-a5d2-310114c98b74\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " Jul 09 00:56:58.800215 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800184 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-oauth-serving-cert\") pod \"85ee6115-e5eb-4f57-a5d2-310114c98b74\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " Jul 09 00:56:58.800350 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800222 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwv5l\" (UniqueName: \"kubernetes.io/projected/85ee6115-e5eb-4f57-a5d2-310114c98b74-kube-api-access-zwv5l\") pod \"85ee6115-e5eb-4f57-a5d2-310114c98b74\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " Jul 09 00:56:58.800350 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800237 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-oauth-config\") pod \"85ee6115-e5eb-4f57-a5d2-310114c98b74\" (UID: \"85ee6115-e5eb-4f57-a5d2-310114c98b74\") " Jul 09 00:56:58.800540 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800512 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-service-ca" (OuterVolumeSpecName: "service-ca") pod "85ee6115-e5eb-4f57-a5d2-310114c98b74" (UID: "85ee6115-e5eb-4f57-a5d2-310114c98b74"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:56:58.800620 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800515 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-config" (OuterVolumeSpecName: "console-config") pod "85ee6115-e5eb-4f57-a5d2-310114c98b74" (UID: "85ee6115-e5eb-4f57-a5d2-310114c98b74"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:56:58.800620 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.800606 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "85ee6115-e5eb-4f57-a5d2-310114c98b74" (UID: "85ee6115-e5eb-4f57-a5d2-310114c98b74"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:56:58.802375 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.802356 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "85ee6115-e5eb-4f57-a5d2-310114c98b74" (UID: "85ee6115-e5eb-4f57-a5d2-310114c98b74"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:56:58.802477 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.802458 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ee6115-e5eb-4f57-a5d2-310114c98b74-kube-api-access-zwv5l" (OuterVolumeSpecName: "kube-api-access-zwv5l") pod "85ee6115-e5eb-4f57-a5d2-310114c98b74" (UID: "85ee6115-e5eb-4f57-a5d2-310114c98b74"). InnerVolumeSpecName "kube-api-access-zwv5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 00:56:58.802516 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.802469 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "85ee6115-e5eb-4f57-a5d2-310114c98b74" (UID: "85ee6115-e5eb-4f57-a5d2-310114c98b74"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:56:58.901714 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.901688 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwv5l\" (UniqueName: \"kubernetes.io/projected/85ee6115-e5eb-4f57-a5d2-310114c98b74-kube-api-access-zwv5l\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:56:58.901714 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.901710 2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-oauth-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:56:58.901842 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.901719 2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:56:58.901842 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.901728 2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:56:58.901842 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.901736 2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-console-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:56:58.901842 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:58.901744 2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee6115-e5eb-4f57-a5d2-310114c98b74-oauth-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:56:59.517139 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.517111 2543 generic.go:356] "Generic (PLEG): container finished" podID="85ee6115-e5eb-4f57-a5d2-310114c98b74" containerID="4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97" exitCode=0 Jul 09 00:56:59.517549 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.517166 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869484c78f-mxxh8" Jul 09 00:56:59.517549 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.517208 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869484c78f-mxxh8" event={"ID":"85ee6115-e5eb-4f57-a5d2-310114c98b74","Type":"ContainerDied","Data":"4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97"} Jul 09 00:56:59.517549 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.517269 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869484c78f-mxxh8" event={"ID":"85ee6115-e5eb-4f57-a5d2-310114c98b74","Type":"ContainerDied","Data":"6e56426a88b9386c67750bd902aca272ade5b915ab530769b565539394459d58"} Jul 09 00:56:59.517549 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.517305 2543 scope.go:117] "RemoveContainer" containerID="4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97" Jul 09 00:56:59.525955 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.525938 2543 scope.go:117] "RemoveContainer" containerID="4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97" Jul 09 00:56:59.526256 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:56:59.526237 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97\": container with ID starting with 4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97 not found: ID does not exist" containerID="4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97" Jul 09 00:56:59.526307 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.526265 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97"} err="failed to get container status \"4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97\": rpc error: code = NotFound desc = could not find container \"4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97\": container with ID starting with 4e1ec818b5996784ba8595f4c2012eb497e31999260b8a838e68d3b5397f6a97 not found: ID does not exist" Jul 09 00:56:59.532080 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.532056 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869484c78f-mxxh8"] Jul 09 00:56:59.535522 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:56:59.535499 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-869484c78f-mxxh8"] Jul 09 00:57:00.927874 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:00.927840 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ee6115-e5eb-4f57-a5d2-310114c98b74" path="/var/lib/kubelet/pods/85ee6115-e5eb-4f57-a5d2-310114c98b74/volumes" Jul 09 00:57:05.726585 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:05.726541 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:05.778428 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:05.778401 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:10.338775 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:10.338744 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7h8np_29569db2-280a-4e93-a06e-0400f7ba321a/serve-healthcheck-canary/0.log" Jul 09 00:57:10.546206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:10.546176 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:12.823780 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:12.823746 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:12.924711 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:12.924684 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:12.971227 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:12.971209 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:17.928196 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:17.928162 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:17.991367 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:17.991343 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:18.477600 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:18.477571 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2mpcm" Jul 09 00:57:18.606222 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:18.606195 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:22.305794 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:22.305759 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:22.356313 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:22.356272 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:26.536018 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:26.535982 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 00:57:26.536425 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:26.536279 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="alertmanager" containerID="cri-o://c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496" gracePeriod=120 Jul 09 00:57:26.536425 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:26.536374 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="prom-label-proxy" containerID="cri-o://7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd" gracePeriod=120 Jul 09 00:57:26.536425 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:26.536400 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-web" containerID="cri-o://41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250" gracePeriod=120 Jul 09 00:57:26.536634 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:26.536438 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy" containerID="cri-o://f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b" gracePeriod=120 Jul 09 00:57:26.536634 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:26.536462 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="config-reloader" containerID="cri-o://54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c" gracePeriod=120 Jul 09 00:57:26.536634 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:26.536360 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-metric" containerID="cri-o://b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed" gracePeriod=120 Jul 09 00:57:27.373753 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.373722 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:27.426923 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.426901 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:27.638442 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638418 2543 generic.go:356] "Generic (PLEG): container finished" podID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerID="7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd" exitCode=0 Jul 09 00:57:27.638442 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638440 2543 generic.go:356] "Generic (PLEG): container finished" podID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerID="f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b" exitCode=0 Jul 09 00:57:27.638442 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638446 2543 generic.go:356] "Generic (PLEG): container finished" podID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerID="54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c" exitCode=0 Jul 09 00:57:27.638442 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638451 2543 generic.go:356] "Generic (PLEG): container finished" podID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerID="c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496" exitCode=0 Jul 09 00:57:27.638904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638499 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd"} Jul 09 00:57:27.638904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638603 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b"} Jul 09 00:57:27.638904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638671 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c"} Jul 09 00:57:27.638904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.638716 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496"} Jul 09 00:57:27.784959 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.784936 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:57:27.822710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822485 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-metrics-client-ca\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.822710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822538 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-trusted-ca-bundle\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.822710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822565 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-main-tls\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.822710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822611 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-cluster-tls-config\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.822710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822637 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.822710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822685 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-volume\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.822710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822719 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822742 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkt4g\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-kube-api-access-hkt4g\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822788 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-out\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822817 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-tls-assets\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822852 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-web\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822924 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-main-db\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822925 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.822966 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-web-config\") pod \"f7f71c45-83e0-40f4-9032-a68e5a91d666\" (UID: \"f7f71c45-83e0-40f4-9032-a68e5a91d666\") " Jul 09 00:57:27.823163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.823006 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:27.823570 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.823224 2543 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-metrics-client-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.823570 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.823242 2543 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.824159 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.824103 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 00:57:27.830960 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.830904 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:27.831345 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.831314 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 00:57:27.831687 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.831553 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-kube-api-access-hkt4g" (OuterVolumeSpecName: "kube-api-access-hkt4g") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "kube-api-access-hkt4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 00:57:27.831829 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.831806 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-out" (OuterVolumeSpecName: "config-out") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 00:57:27.832439 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.832378 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-volume" (OuterVolumeSpecName: "config-volume") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:27.833781 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.833755 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:27.834238 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.834189 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:27.834238 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.834147 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:27.835078 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.835055 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:27.841857 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.841834 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-web-config" (OuterVolumeSpecName: "web-config") pod "f7f71c45-83e0-40f4-9032-a68e5a91d666" (UID: "f7f71c45-83e0-40f4-9032-a68e5a91d666"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:27.924483 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924427 2543 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-alertmanager-main-db\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924483 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924470 2543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-web-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924483 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924479 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-main-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924488 2543 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-cluster-tls-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924498 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924507 2543 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-volume\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924517 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924528 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkt4g\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-kube-api-access-hkt4g\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924535 2543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7f71c45-83e0-40f4-9032-a68e5a91d666-config-out\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924543 2543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7f71c45-83e0-40f4-9032-a68e5a91d666-tls-assets\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:27.924670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:27.924552 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7f71c45-83e0-40f4-9032-a68e5a91d666-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:28.645308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.645280 2543 generic.go:356] "Generic (PLEG): container finished" podID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerID="b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed" exitCode=0 Jul 09 00:57:28.645308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.645308 2543 generic.go:356] "Generic (PLEG): container finished" podID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerID="41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250" exitCode=0 Jul 09 00:57:28.645633 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.645352 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 00:57:28.645633 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.645350 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed"} Jul 09 00:57:28.645633 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.645492 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250"} Jul 09 00:57:28.645633 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.645523 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7f71c45-83e0-40f4-9032-a68e5a91d666","Type":"ContainerDied","Data":"d06e807296333af799558250cfdeb00565592ad3b359005e8820e6f6b8a30ad6"} Jul 09 00:57:28.645633 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.645552 2543 scope.go:117] "RemoveContainer" containerID="7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd" Jul 09 00:57:28.653531 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.653373 2543 scope.go:117] "RemoveContainer" containerID="b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed" Jul 09 00:57:28.664094 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.664071 2543 scope.go:117] "RemoveContainer" containerID="f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b" Jul 09 00:57:28.671502 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.671480 2543 scope.go:117] "RemoveContainer" containerID="41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250" Jul 09 00:57:28.671916 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.671895 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 00:57:28.675415 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.675396 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 00:57:28.678748 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.678729 2543 scope.go:117] "RemoveContainer" containerID="54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c" Jul 09 00:57:28.685336 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.685317 2543 scope.go:117] "RemoveContainer" containerID="c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496" Jul 09 00:57:28.692078 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.692042 2543 scope.go:117] "RemoveContainer" containerID="cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85" Jul 09 00:57:28.698672 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.698652 2543 scope.go:117] "RemoveContainer" containerID="7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd" Jul 09 00:57:28.698970 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:28.698942 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd\": container with ID starting with 7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd not found: ID does not exist" containerID="7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd" Jul 09 00:57:28.699050 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.698980 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd"} err="failed to get container status \"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd\": rpc error: code = NotFound desc = could not find container \"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd\": container with ID starting with 7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd not found: ID does not exist" Jul 09 00:57:28.699050 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.699005 2543 scope.go:117] "RemoveContainer" containerID="b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed" Jul 09 00:57:28.699258 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:28.699211 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed\": container with ID starting with b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed not found: ID does not exist" containerID="b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed" Jul 09 00:57:28.699258 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.699240 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed"} err="failed to get container status \"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed\": rpc error: code = NotFound desc = could not find container \"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed\": container with ID starting with b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed not found: ID does not exist" Jul 09 00:57:28.699374 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.699263 2543 scope.go:117] "RemoveContainer" containerID="f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b" Jul 09 00:57:28.699659 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:28.699618 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b\": container with ID starting with f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b not found: ID does not exist" containerID="f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b" Jul 09 00:57:28.699778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.699669 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b"} err="failed to get container status \"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b\": rpc error: code = NotFound desc = could not find container \"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b\": container with ID starting with f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b not found: ID does not exist" Jul 09 00:57:28.699778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.699689 2543 scope.go:117] "RemoveContainer" containerID="41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250" Jul 09 00:57:28.700008 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:28.699989 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250\": container with ID starting with 41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250 not found: ID does not exist" containerID="41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250" Jul 09 00:57:28.700058 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700012 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250"} err="failed to get container status \"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250\": rpc error: code = NotFound desc = could not find container \"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250\": container with ID starting with 41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250 not found: ID does not exist" Jul 09 00:57:28.700058 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700026 2543 scope.go:117] "RemoveContainer" containerID="54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c" Jul 09 00:57:28.700278 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:28.700261 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c\": container with ID starting with 54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c not found: ID does not exist" containerID="54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c" Jul 09 00:57:28.700324 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700282 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c"} err="failed to get container status \"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c\": rpc error: code = NotFound desc = could not find container \"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c\": container with ID starting with 54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c not found: ID does not exist" Jul 09 00:57:28.700324 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700297 2543 scope.go:117] "RemoveContainer" containerID="c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496" Jul 09 00:57:28.700515 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:28.700503 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496\": container with ID starting with c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496 not found: ID does not exist" containerID="c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496" Jul 09 00:57:28.700556 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700517 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496"} err="failed to get container status \"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496\": rpc error: code = NotFound desc = could not find container \"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496\": container with ID starting with c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496 not found: ID does not exist" Jul 09 00:57:28.700556 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700528 2543 scope.go:117] "RemoveContainer" containerID="cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85" Jul 09 00:57:28.700782 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:28.700764 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85\": container with ID starting with cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85 not found: ID does not exist" containerID="cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85" Jul 09 00:57:28.700837 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700787 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85"} err="failed to get container status \"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85\": rpc error: code = NotFound desc = could not find container \"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85\": container with ID starting with cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85 not found: ID does not exist" Jul 09 00:57:28.700837 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.700802 2543 scope.go:117] "RemoveContainer" containerID="7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd" Jul 09 00:57:28.701042 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701025 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd"} err="failed to get container status \"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd\": rpc error: code = NotFound desc = could not find container \"7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd\": container with ID starting with 7d4e916ed4052a033bda6c257977dc558a434d5371be037da6632786c7cf97dd not found: ID does not exist" Jul 09 00:57:28.701119 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701043 2543 scope.go:117] "RemoveContainer" containerID="b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed" Jul 09 00:57:28.701244 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701230 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed"} err="failed to get container status \"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed\": rpc error: code = NotFound desc = could not find container \"b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed\": container with ID starting with b345cf3e6f2d28a0e9b02c539a15c1c13a03ac0bf647171a1a0b8519bb153fed not found: ID does not exist" Jul 09 00:57:28.701286 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701246 2543 scope.go:117] "RemoveContainer" containerID="f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b" Jul 09 00:57:28.701452 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701434 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b"} err="failed to get container status \"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b\": rpc error: code = NotFound desc = could not find container \"f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b\": container with ID starting with f1760558e7bea05fe45d5bf6123e63041aa1a7c00e94108a1b5e382928b2cf9b not found: ID does not exist" Jul 09 00:57:28.701517 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701455 2543 scope.go:117] "RemoveContainer" containerID="41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250" Jul 09 00:57:28.701714 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701697 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250"} err="failed to get container status \"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250\": rpc error: code = NotFound desc = could not find container \"41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250\": container with ID starting with 41d32555ded941c5f57f908edab4662364ac50b8aa6e2972e369ec6a60f03250 not found: ID does not exist" Jul 09 00:57:28.701767 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701714 2543 scope.go:117] "RemoveContainer" containerID="54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c" Jul 09 00:57:28.701919 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701896 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c"} err="failed to get container status \"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c\": rpc error: code = NotFound desc = could not find container \"54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c\": container with ID starting with 54b57132b357fdf1477943de2d6f707a93d34aae5a7fc5b3427c7b3b88ca4f1c not found: ID does not exist" Jul 09 00:57:28.701959 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.701920 2543 scope.go:117] "RemoveContainer" containerID="c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496" Jul 09 00:57:28.702098 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.702080 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496"} err="failed to get container status \"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496\": rpc error: code = NotFound desc = could not find container \"c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496\": container with ID starting with c5a92391ac6ea00b4e99f6cf1d1f4d59d082e483bce3f1b3fb1ea4d60d381496 not found: ID does not exist" Jul 09 00:57:28.702098 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.702095 2543 scope.go:117] "RemoveContainer" containerID="cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85" Jul 09 00:57:28.702283 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.702265 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85"} err="failed to get container status \"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85\": rpc error: code = NotFound desc = could not find container \"cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85\": container with ID starting with cbf2d25db2ef78f5c8394758c793b91ca5ac2a9052925ff7ad3901b0493f7b85 not found: ID does not exist" Jul 09 00:57:28.928104 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:28.928032 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" path="/var/lib/kubelet/pods/f7f71c45-83e0-40f4-9032-a68e5a91d666/volumes" Jul 09 00:57:30.743724 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:30.743691 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:57:30.744319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:30.743981 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="prometheus" containerID="cri-o://f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed" gracePeriod=600 Jul 09 00:57:30.744319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:30.743994 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy" containerID="cri-o://a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9" gracePeriod=600 Jul 09 00:57:30.744319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:30.744013 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="thanos-sidecar" containerID="cri-o://9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb" gracePeriod=600 Jul 09 00:57:30.744319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:30.744032 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="config-reloader" containerID="cri-o://dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a" gracePeriod=600 Jul 09 00:57:30.744319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:30.744085 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223" gracePeriod=600 Jul 09 00:57:30.744566 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:30.744105 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-web" containerID="cri-o://0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694" gracePeriod=600 Jul 09 00:57:31.659680 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659631 2543 generic.go:356] "Generic (PLEG): container finished" podID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerID="b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223" exitCode=0 Jul 09 00:57:31.659680 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659675 2543 generic.go:356] "Generic (PLEG): container finished" podID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerID="a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9" exitCode=0 Jul 09 00:57:31.659680 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659685 2543 generic.go:356] "Generic (PLEG): container finished" podID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerID="9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb" exitCode=0 Jul 09 00:57:31.659904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659695 2543 generic.go:356] "Generic (PLEG): container finished" podID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerID="dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a" exitCode=0 Jul 09 00:57:31.659904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659702 2543 generic.go:356] "Generic (PLEG): container finished" podID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerID="f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed" exitCode=0 Jul 09 00:57:31.659904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659706 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223"} Jul 09 00:57:31.659904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659769 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9"} Jul 09 00:57:31.659904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659795 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb"} Jul 09 00:57:31.659904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659819 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a"} Jul 09 00:57:31.659904 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.659842 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed"} Jul 09 00:57:31.986161 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:31.986135 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.054628 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.054600 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-metrics-client-ca\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.054628 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.054636 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-487n4\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-kube-api-access-487n4\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.054836 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.054674 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-config-out\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.054896 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.054865 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.054950 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.054908 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-trusted-ca-bundle\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055000 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.054949 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-db\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055050 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055008 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-tls-assets\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055050 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055035 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055149 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055064 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-config\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055149 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055097 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-2\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055149 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055127 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-0\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055289 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055161 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-metrics-client-certs\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055289 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055196 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-web-config\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055289 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055223 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-kube-rbac-proxy\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055289 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055274 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-grpc-tls\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055499 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055314 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-1\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055499 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055348 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-tls\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055499 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055374 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-thanos-prometheus-http-client-file\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055499 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055414 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-serving-certs-ca-bundle\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055499 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055441 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-kubelet-serving-ca-bundle\") pod \"8e41e0a6-e9b0-4163-ad61-1586ab460586\" (UID: \"8e41e0a6-e9b0-4163-ad61-1586ab460586\") " Jul 09 00:57:32.055760 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055028 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:32.055760 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055727 2543 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-metrics-client-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.055760 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055437 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:32.055934 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.055910 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:32.057630 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.057314 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:32.057630 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.057542 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:32.058504 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.058475 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.058619 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.058588 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:32.059833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.059793 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 00:57:32.059940 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.059912 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:57:32.060134 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.060110 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 00:57:32.060673 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.060527 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-kube-api-access-487n4" (OuterVolumeSpecName: "kube-api-access-487n4") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "kube-api-access-487n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 00:57:32.060673 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.060532 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-config-out" (OuterVolumeSpecName: "config-out") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 00:57:32.060673 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.060531 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.060866 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.060748 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.060924 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.060877 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-config" (OuterVolumeSpecName: "config") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.060981 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.060958 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.061092 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.061019 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.061092 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.061066 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.062160 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.062069 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.070280 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.070259 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-web-config" (OuterVolumeSpecName: "web-config") pod "8e41e0a6-e9b0-4163-ad61-1586ab460586" (UID: "8e41e0a6-e9b0-4163-ad61-1586ab460586"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:57:32.156484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156455 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156480 2543 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-metrics-client-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156489 2543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-web-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156499 2543 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-kube-rbac-proxy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156508 2543 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-grpc-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156516 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156527 2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156536 2543 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156545 2543 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156553 2543 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156561 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487n4\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-kube-api-access-487n4\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156568 2543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-config-out\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156576 2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156585 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156595 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-db\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156603 2543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e41e0a6-e9b0-4163-ad61-1586ab460586-tls-assets\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156611 2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156619 2543 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e41e0a6-e9b0-4163-ad61-1586ab460586-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.156637 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.156628 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e41e0a6-e9b0-4163-ad61-1586ab460586-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:57:32.666009 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.665974 2543 generic.go:356] "Generic (PLEG): container finished" podID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerID="0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694" exitCode=0 Jul 09 00:57:32.666169 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.666016 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694"} Jul 09 00:57:32.666169 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.666052 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.666169 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.666088 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e41e0a6-e9b0-4163-ad61-1586ab460586","Type":"ContainerDied","Data":"bf3f9061291e78a4a8ccbb326331f1ed9a98bb4a93be4be19162d23e85cf1e88"} Jul 09 00:57:32.666169 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.666122 2543 scope.go:117] "RemoveContainer" containerID="b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223" Jul 09 00:57:32.677435 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.677419 2543 scope.go:117] "RemoveContainer" containerID="a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9" Jul 09 00:57:32.684423 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.684408 2543 scope.go:117] "RemoveContainer" containerID="0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694" Jul 09 00:57:32.690739 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.690626 2543 scope.go:117] "RemoveContainer" containerID="9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb" Jul 09 00:57:32.696048 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.696026 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:57:32.697535 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.697516 2543 scope.go:117] "RemoveContainer" containerID="dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a" Jul 09 00:57:32.700552 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.700532 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:57:32.704823 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.704804 2543 scope.go:117] "RemoveContainer" containerID="f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed" Jul 09 00:57:32.711375 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.711356 2543 scope.go:117] "RemoveContainer" containerID="186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0" Jul 09 00:57:32.719456 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.719436 2543 scope.go:117] "RemoveContainer" containerID="b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223" Jul 09 00:57:32.719790 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:32.719763 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223\": container with ID starting with b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223 not found: ID does not exist" containerID="b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223" Jul 09 00:57:32.719885 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.719800 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223"} err="failed to get container status \"b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223\": rpc error: code = NotFound desc = could not find container \"b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223\": container with ID starting with b28af160be975030fd9b0ce3a6f13d02f92c299f882bb522fac6c2a49bb11223 not found: ID does not exist" Jul 09 00:57:32.719885 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.719827 2543 scope.go:117] "RemoveContainer" containerID="a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9" Jul 09 00:57:32.720059 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:32.720040 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9\": container with ID starting with a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9 not found: ID does not exist" containerID="a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9" Jul 09 00:57:32.720118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720068 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9"} err="failed to get container status \"a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9\": rpc error: code = NotFound desc = could not find container \"a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9\": container with ID starting with a9a206648dc0d69e522d2ca533d2379e927637df52c0c33d254373ebfc2703d9 not found: ID does not exist" Jul 09 00:57:32.720118 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720091 2543 scope.go:117] "RemoveContainer" containerID="0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694" Jul 09 00:57:32.720337 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:32.720321 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694\": container with ID starting with 0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694 not found: ID does not exist" containerID="0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694" Jul 09 00:57:32.720400 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720343 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694"} err="failed to get container status \"0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694\": rpc error: code = NotFound desc = could not find container \"0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694\": container with ID starting with 0d54ba61d76b94104a941ba21dfdfd028c97d90295d09dd007a110af007fd694 not found: ID does not exist" Jul 09 00:57:32.720400 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720356 2543 scope.go:117] "RemoveContainer" containerID="9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb" Jul 09 00:57:32.720611 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:32.720593 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb\": container with ID starting with 9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb not found: ID does not exist" containerID="9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb" Jul 09 00:57:32.720678 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720618 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb"} err="failed to get container status \"9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb\": rpc error: code = NotFound desc = could not find container \"9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb\": container with ID starting with 9272f0568c7a3dc298e1c88bb5fa65d1db1a31b93bd738e2c55948f9908513cb not found: ID does not exist" Jul 09 00:57:32.720678 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720636 2543 scope.go:117] "RemoveContainer" containerID="dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a" Jul 09 00:57:32.720890 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:32.720874 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a\": container with ID starting with dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a not found: ID does not exist" containerID="dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a" Jul 09 00:57:32.720933 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720895 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a"} err="failed to get container status \"dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a\": rpc error: code = NotFound desc = could not find container \"dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a\": container with ID starting with dfe55b333c72a957f29c7864d0fe13b9114f9872dbb5c160ec02466bd548d60a not found: ID does not exist" Jul 09 00:57:32.720933 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.720908 2543 scope.go:117] "RemoveContainer" containerID="f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed" Jul 09 00:57:32.721152 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:32.721138 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed\": container with ID starting with f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed not found: ID does not exist" containerID="f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed" Jul 09 00:57:32.721192 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.721156 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed"} err="failed to get container status \"f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed\": rpc error: code = NotFound desc = could not find container \"f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed\": container with ID starting with f5d6e7cc2a987a2ce36ed535f9e34c94b84e5af752367c0210ca823f271044ed not found: ID does not exist" Jul 09 00:57:32.721192 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.721169 2543 scope.go:117] "RemoveContainer" containerID="186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0" Jul 09 00:57:32.721385 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:57:32.721370 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0\": container with ID starting with 186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0 not found: ID does not exist" containerID="186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0" Jul 09 00:57:32.721431 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.721387 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0"} err="failed to get container status \"186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0\": rpc error: code = NotFound desc = could not find container \"186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0\": container with ID starting with 186ae24f0986c77d287333a3eb3b6c7f6539af7d043eb4fbd97bcde9dffa9fc0 not found: ID does not exist" Jul 09 00:57:32.723936 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.723916 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:57:32.724182 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724171 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="prometheus" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724184 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="prometheus" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724193 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="config-reloader" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724198 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="config-reloader" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724205 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="alertmanager" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724212 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="alertmanager" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724218 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-metric" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724224 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-metric" Jul 09 00:57:32.724231 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724230 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-web" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724235 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-web" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724241 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="prom-label-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724247 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="prom-label-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724253 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ee6115-e5eb-4f57-a5d2-310114c98b74" containerName="console" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724258 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ee6115-e5eb-4f57-a5d2-310114c98b74" containerName="console" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724264 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="init-config-reloader" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724269 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="init-config-reloader" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724276 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="thanos-sidecar" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724282 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="thanos-sidecar" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724292 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724297 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724303 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="init-config-reloader" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724307 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="init-config-reloader" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724313 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-thanos" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724318 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-thanos" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724324 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-web" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724329 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-web" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724334 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="config-reloader" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724339 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="config-reloader" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724344 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724350 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724390 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="prometheus" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724396 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="prom-label-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724401 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ee6115-e5eb-4f57-a5d2-310114c98b74" containerName="console" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724407 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724412 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="config-reloader" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724417 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-web" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724423 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="alertmanager" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724427 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy-thanos" Jul 09 00:57:32.724475 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724433 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="config-reloader" Jul 09 00:57:32.725286 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724437 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="thanos-sidecar" Jul 09 00:57:32.725286 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724442 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-web" Jul 09 00:57:32.725286 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724448 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" containerName="kube-rbac-proxy" Jul 09 00:57:32.725286 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.724455 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f71c45-83e0-40f4-9032-a68e5a91d666" containerName="kube-rbac-proxy-metric" Jul 09 00:57:32.728833 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.728813 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.731356 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.731336 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 00:57:32.731470 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.731414 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 00:57:32.732022 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.731961 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2hz55\"" Jul 09 00:57:32.732022 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.732019 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 00:57:32.732630 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.732605 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 00:57:32.732749 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.732714 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 00:57:32.732829 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.732802 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 00:57:32.732897 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.732629 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 00:57:32.732966 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.732948 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 00:57:32.733058 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.733041 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 00:57:32.733125 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.733094 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 00:57:32.733228 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.733114 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 00:57:32.733462 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.733446 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 00:57:32.733700 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.733539 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-41ltc3jf8fgj0\"" Jul 09 00:57:32.733700 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.733658 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 00:57:32.735556 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.735535 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 00:57:32.737374 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.737350 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 00:57:32.742710 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.742691 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:57:32.761626 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761599 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761788 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761632 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761788 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761671 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d598fcf-d7bb-4a28-a559-c726182008ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761788 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761690 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761788 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761758 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761965 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761805 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761965 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761835 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761965 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761873 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-config\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.761965 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761916 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkwv\" (UniqueName: \"kubernetes.io/projected/0d598fcf-d7bb-4a28-a559-c726182008ea-kube-api-access-zjkwv\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762100 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.761988 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762100 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762016 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762100 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762034 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762100 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762053 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762100 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762087 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762130 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d598fcf-d7bb-4a28-a559-c726182008ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762156 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762186 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762226 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762247 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.762308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.762265 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863158 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863123 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863158 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863160 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863379 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863177 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863379 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863310 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863379 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863351 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863533 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863401 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d598fcf-d7bb-4a28-a559-c726182008ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863533 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863434 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863533 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863470 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863533 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863501 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863533 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863526 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863549 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863575 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863623 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863683 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d598fcf-d7bb-4a28-a559-c726182008ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863711 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863755 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863781 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.863813 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863806 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.864168 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863843 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-config\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.864168 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.863870 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkwv\" (UniqueName: \"kubernetes.io/projected/0d598fcf-d7bb-4a28-a559-c726182008ea-kube-api-access-zjkwv\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.864387 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.864361 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.864767 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.864742 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.865369 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.865342 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.866226 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.866200 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d598fcf-d7bb-4a28-a559-c726182008ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.866428 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.866303 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.866428 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.866306 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.866599 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.866505 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.867295 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.867024 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.867295 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.867057 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.867295 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.867253 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.867487 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.867309 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.867778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.867713 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.867990 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.867963 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d598fcf-d7bb-4a28-a559-c726182008ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.868070 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.868041 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.868670 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.868625 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.868743 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.868695 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.868862 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.868846 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-config\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.869264 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.869249 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d598fcf-d7bb-4a28-a559-c726182008ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.869872 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.869855 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d598fcf-d7bb-4a28-a559-c726182008ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.872078 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.872055 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkwv\" (UniqueName: \"kubernetes.io/projected/0d598fcf-d7bb-4a28-a559-c726182008ea-kube-api-access-zjkwv\") pod \"prometheus-k8s-0\" (UID: \"0d598fcf-d7bb-4a28-a559-c726182008ea\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:32.927991 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:32.927918 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e41e0a6-e9b0-4163-ad61-1586ab460586" path="/var/lib/kubelet/pods/8e41e0a6-e9b0-4163-ad61-1586ab460586/volumes" Jul 09 00:57:33.040745 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:33.040694 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:33.172086 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:33.172051 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 00:57:33.174678 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:57:33.174636 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d598fcf_d7bb_4a28_a559_c726182008ea.slice/crio-9f465a5fb0eea9d5d60cd8f9ad495d7a3508ac6fecc54c66f6c2a835c75100ac WatchSource:0}: Error finding container 9f465a5fb0eea9d5d60cd8f9ad495d7a3508ac6fecc54c66f6c2a835c75100ac: Status 404 returned error can't find the container with id 9f465a5fb0eea9d5d60cd8f9ad495d7a3508ac6fecc54c66f6c2a835c75100ac Jul 09 00:57:33.675821 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:33.675786 2543 generic.go:356] "Generic (PLEG): container finished" podID="0d598fcf-d7bb-4a28-a559-c726182008ea" containerID="e3732d156d474b1c48d2a964078323048ec2546a8d2dbcaa828d55fa2e99c0d0" exitCode=0 Jul 09 00:57:33.676011 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:33.675877 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerDied","Data":"e3732d156d474b1c48d2a964078323048ec2546a8d2dbcaa828d55fa2e99c0d0"} Jul 09 00:57:33.676011 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:33.675988 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerStarted","Data":"9f465a5fb0eea9d5d60cd8f9ad495d7a3508ac6fecc54c66f6c2a835c75100ac"} Jul 09 00:57:34.683689 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:34.683634 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerStarted","Data":"57628aa2e42783ac322a24fa615cdecc98d4d1920d8fc8ff84ed8d3b4ac33b02"} Jul 09 00:57:34.684127 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:34.683720 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerStarted","Data":"e18e3d03d94b115678a273921f4ae5332b39b1108f5a992902c1b51c50c65f24"} Jul 09 00:57:34.684127 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:34.683747 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerStarted","Data":"e24434ed8be696d5a49c4a3b45ba698788c926e18d25a13edeb499f48e1d2783"} Jul 09 00:57:34.684127 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:34.683772 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerStarted","Data":"aaabaea3f5fb45e60f4565de582495eeed8573b34791570dbd4a915945093273"} Jul 09 00:57:34.684127 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:34.683795 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerStarted","Data":"b1bfd7ceb9b077d4b976f9374088011d54be40d12ecc87f786f8913a4fc8f227"} Jul 09 00:57:34.684127 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:34.683818 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d598fcf-d7bb-4a28-a559-c726182008ea","Type":"ContainerStarted","Data":"4b893734a77356e3fb9ead03b7242d88d5107e00481b3d04c33f610f08748428"} Jul 09 00:57:34.712227 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:34.712168 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6909441640000002 podStartE2EDuration="2.712147953s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.203789ms" isStatefulPod=false podCreationTimestamp="2026-07-09 00:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:57:34.711350552 +0000 UTC m=+118.989189241" watchObservedRunningTime="2026-07-09 00:57:34.712147953 +0000 UTC m=+118.989986604" Jul 09 00:57:35.726815 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:35.726783 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:35.824634 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:35.824602 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:35.924906 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:35.924879 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:38.040848 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:38.040816 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:57:48.030734 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:48.030691 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:48.078924 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:48.078892 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:55.959221 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:55.959186 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:58.461206 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:58.461165 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8578cb8667-gxccm"] Jul 09 00:57:58.557819 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:58.557786 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:58.659887 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:58.659857 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:57:58.715907 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:57:58.715812 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:02.074718 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:02.074681 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:02.185947 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:02.185915 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:23.481958 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.481894 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8578cb8667-gxccm" podUID="326496df-f7b7-4eff-a9c4-b8294cbeaca6" containerName="console" containerID="cri-o://995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd" gracePeriod=15 Jul 09 00:58:23.720019 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.719998 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:58:23.759723 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.759669 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-serving-cert\") pod \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " Jul 09 00:58:23.759723 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.759709 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-oauth-serving-cert\") pod \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " Jul 09 00:58:23.759870 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.759727 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-oauth-config\") pod \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " Jul 09 00:58:23.759870 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.759759 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-trusted-ca-bundle\") pod \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " Jul 09 00:58:23.759870 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.759812 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzmhj\" (UniqueName: \"kubernetes.io/projected/326496df-f7b7-4eff-a9c4-b8294cbeaca6-kube-api-access-lzmhj\") pod \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " Jul 09 00:58:23.760018 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.759868 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-config\") pod \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " Jul 09 00:58:23.760018 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.759892 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-service-ca\") pod \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\" (UID: \"326496df-f7b7-4eff-a9c4-b8294cbeaca6\") " Jul 09 00:58:23.760205 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.760179 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "326496df-f7b7-4eff-a9c4-b8294cbeaca6" (UID: "326496df-f7b7-4eff-a9c4-b8294cbeaca6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:58:23.760294 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.760261 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "326496df-f7b7-4eff-a9c4-b8294cbeaca6" (UID: "326496df-f7b7-4eff-a9c4-b8294cbeaca6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:58:23.760350 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.760303 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-config" (OuterVolumeSpecName: "console-config") pod "326496df-f7b7-4eff-a9c4-b8294cbeaca6" (UID: "326496df-f7b7-4eff-a9c4-b8294cbeaca6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:58:23.760393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.760344 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-service-ca" (OuterVolumeSpecName: "service-ca") pod "326496df-f7b7-4eff-a9c4-b8294cbeaca6" (UID: "326496df-f7b7-4eff-a9c4-b8294cbeaca6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 00:58:23.761791 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.761772 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "326496df-f7b7-4eff-a9c4-b8294cbeaca6" (UID: "326496df-f7b7-4eff-a9c4-b8294cbeaca6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:58:23.762296 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.762270 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "326496df-f7b7-4eff-a9c4-b8294cbeaca6" (UID: "326496df-f7b7-4eff-a9c4-b8294cbeaca6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 00:58:23.762296 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.762282 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326496df-f7b7-4eff-a9c4-b8294cbeaca6-kube-api-access-lzmhj" (OuterVolumeSpecName: "kube-api-access-lzmhj") pod "326496df-f7b7-4eff-a9c4-b8294cbeaca6" (UID: "326496df-f7b7-4eff-a9c4-b8294cbeaca6"). InnerVolumeSpecName "kube-api-access-lzmhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 00:58:23.860882 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.860860 2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:58:23.860882 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.860881 2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-oauth-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:58:23.861006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.860890 2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-oauth-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:58:23.861006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.860899 2543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-trusted-ca-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:58:23.861006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.860909 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzmhj\" (UniqueName: \"kubernetes.io/projected/326496df-f7b7-4eff-a9c4-b8294cbeaca6-kube-api-access-lzmhj\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:58:23.861006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.860921 2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-console-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:58:23.861006 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.860929 2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/326496df-f7b7-4eff-a9c4-b8294cbeaca6-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 00:58:23.866831 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.866807 2543 generic.go:356] "Generic (PLEG): container finished" podID="326496df-f7b7-4eff-a9c4-b8294cbeaca6" containerID="995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd" exitCode=0 Jul 09 00:58:23.866915 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.866858 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8578cb8667-gxccm" Jul 09 00:58:23.866978 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.866899 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578cb8667-gxccm" event={"ID":"326496df-f7b7-4eff-a9c4-b8294cbeaca6","Type":"ContainerDied","Data":"995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd"} Jul 09 00:58:23.866978 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.866960 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8578cb8667-gxccm" event={"ID":"326496df-f7b7-4eff-a9c4-b8294cbeaca6","Type":"ContainerDied","Data":"8d1424fe50c6f6f27609a701fba99e397e25d92a1a0a8a10eade14b35c38a3a9"} Jul 09 00:58:23.867073 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.866996 2543 scope.go:117] "RemoveContainer" containerID="995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd" Jul 09 00:58:23.879454 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.875491 2543 scope.go:117] "RemoveContainer" containerID="995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd" Jul 09 00:58:23.879454 ip-10-0-134-128 kubenswrapper[2543]: E0709 00:58:23.876999 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd\": container with ID starting with 995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd not found: ID does not exist" containerID="995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd" Jul 09 00:58:23.879454 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.877038 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd"} err="failed to get container status \"995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd\": rpc error: code = NotFound desc = could not find container \"995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd\": container with ID starting with 995464a1ca829e73ca50e88cfebab097931036be1fbf6fa4c340f503417353bd not found: ID does not exist" Jul 09 00:58:23.887307 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.887283 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8578cb8667-gxccm"] Jul 09 00:58:23.890942 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:23.890922 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8578cb8667-gxccm"] Jul 09 00:58:24.927608 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:24.927573 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326496df-f7b7-4eff-a9c4-b8294cbeaca6" path="/var/lib/kubelet/pods/326496df-f7b7-4eff-a9c4-b8294cbeaca6/volumes" Jul 09 00:58:28.533158 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:28.533079 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:29.885997 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:29.885933 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:32.143837 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:32.143805 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:32.244371 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:32.244337 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:33.041037 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:33.041004 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:58:33.056500 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:33.056474 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:58:33.916350 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:33.916324 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 00:58:33.952538 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:33.952508 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:34.156635 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:34.156603 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:34.345523 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:34.345492 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:34.449973 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:34.449935 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:36.525283 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:36.525253 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:36.909401 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:36.909332 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:39.130878 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:39.130842 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:39.231244 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:39.231212 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:40.481726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:40.481692 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:40.581319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:40.581290 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:41.699347 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:41.699310 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:41.752233 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:41.752202 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:44.518336 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.518305 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:44.622264 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.622241 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bg2z2"] Jul 09 00:58:44.622507 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.622497 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326496df-f7b7-4eff-a9c4-b8294cbeaca6" containerName="console" Jul 09 00:58:44.622550 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.622511 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="326496df-f7b7-4eff-a9c4-b8294cbeaca6" containerName="console" Jul 09 00:58:44.622583 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.622564 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="326496df-f7b7-4eff-a9c4-b8294cbeaca6" containerName="console" Jul 09 00:58:44.626706 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.626692 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:44.629144 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.629126 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 00:58:44.629261 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.629129 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wflbs\"" Jul 09 00:58:44.629863 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.629848 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 00:58:44.632267 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.632247 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bg2z2"] Jul 09 00:58:44.709683 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.709633 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/883007c2-5de5-4310-ae50-fd129c1f53fe-bound-sa-token\") pod \"cert-manager-55b4d954d5-bg2z2\" (UID: \"883007c2-5de5-4310-ae50-fd129c1f53fe\") " pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:44.709811 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.709755 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h42c\" (UniqueName: \"kubernetes.io/projected/883007c2-5de5-4310-ae50-fd129c1f53fe-kube-api-access-5h42c\") pod \"cert-manager-55b4d954d5-bg2z2\" (UID: \"883007c2-5de5-4310-ae50-fd129c1f53fe\") " pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:44.810595 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.810524 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/883007c2-5de5-4310-ae50-fd129c1f53fe-bound-sa-token\") pod \"cert-manager-55b4d954d5-bg2z2\" (UID: \"883007c2-5de5-4310-ae50-fd129c1f53fe\") " pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:44.810731 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.810601 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h42c\" (UniqueName: \"kubernetes.io/projected/883007c2-5de5-4310-ae50-fd129c1f53fe-kube-api-access-5h42c\") pod \"cert-manager-55b4d954d5-bg2z2\" (UID: \"883007c2-5de5-4310-ae50-fd129c1f53fe\") " pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:44.817911 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.817881 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/883007c2-5de5-4310-ae50-fd129c1f53fe-bound-sa-token\") pod \"cert-manager-55b4d954d5-bg2z2\" (UID: \"883007c2-5de5-4310-ae50-fd129c1f53fe\") " pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:44.818007 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.817969 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h42c\" (UniqueName: \"kubernetes.io/projected/883007c2-5de5-4310-ae50-fd129c1f53fe-kube-api-access-5h42c\") pod \"cert-manager-55b4d954d5-bg2z2\" (UID: \"883007c2-5de5-4310-ae50-fd129c1f53fe\") " pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:44.945787 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:44.945764 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bg2z2" Jul 09 00:58:45.067981 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:45.067952 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bg2z2"] Jul 09 00:58:45.070517 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:58:45.070485 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod883007c2_5de5_4310_ae50_fd129c1f53fe.slice/crio-0f358e56dbf8af3b373c5985936acacb54b381224841db39de4e031f7c3e7ce1 WatchSource:0}: Error finding container 0f358e56dbf8af3b373c5985936acacb54b381224841db39de4e031f7c3e7ce1: Status 404 returned error can't find the container with id 0f358e56dbf8af3b373c5985936acacb54b381224841db39de4e031f7c3e7ce1 Jul 09 00:58:45.942924 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:45.942883 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bg2z2" event={"ID":"883007c2-5de5-4310-ae50-fd129c1f53fe","Type":"ContainerStarted","Data":"0f358e56dbf8af3b373c5985936acacb54b381224841db39de4e031f7c3e7ce1"} Jul 09 00:58:46.835138 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:46.835096 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:46.933147 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:46.933115 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:48.478004 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.477975 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv"] Jul 09 00:58:48.480995 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.480979 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.483183 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.483161 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7tk4r\"" Jul 09 00:58:48.483301 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.483180 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 00:58:48.484140 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.484125 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 00:58:48.488491 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.488472 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv"] Jul 09 00:58:48.546433 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.546408 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89f5\" (UniqueName: \"kubernetes.io/projected/e367f5d8-b32d-46ec-8202-72db8ca2323d-kube-api-access-b89f5\") pod \"openshift-lws-operator-fd8ccff4c-x6zwv\" (UID: \"e367f5d8-b32d-46ec-8202-72db8ca2323d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.546536 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.546442 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e367f5d8-b32d-46ec-8202-72db8ca2323d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x6zwv\" (UID: \"e367f5d8-b32d-46ec-8202-72db8ca2323d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.647106 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.647052 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89f5\" (UniqueName: \"kubernetes.io/projected/e367f5d8-b32d-46ec-8202-72db8ca2323d-kube-api-access-b89f5\") pod \"openshift-lws-operator-fd8ccff4c-x6zwv\" (UID: \"e367f5d8-b32d-46ec-8202-72db8ca2323d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.647106 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.647080 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e367f5d8-b32d-46ec-8202-72db8ca2323d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x6zwv\" (UID: \"e367f5d8-b32d-46ec-8202-72db8ca2323d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.647363 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.647348 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e367f5d8-b32d-46ec-8202-72db8ca2323d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x6zwv\" (UID: \"e367f5d8-b32d-46ec-8202-72db8ca2323d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.656535 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.656511 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89f5\" (UniqueName: \"kubernetes.io/projected/e367f5d8-b32d-46ec-8202-72db8ca2323d-kube-api-access-b89f5\") pod \"openshift-lws-operator-fd8ccff4c-x6zwv\" (UID: \"e367f5d8-b32d-46ec-8202-72db8ca2323d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.790007 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.789982 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" Jul 09 00:58:48.914026 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.913847 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv"] Jul 09 00:58:48.916164 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:58:48.916135 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode367f5d8_b32d_46ec_8202_72db8ca2323d.slice/crio-2b10c4b2205921a8a16eb031ffbeb69a1b4126e5cee7a4ebdddd761110b37970 WatchSource:0}: Error finding container 2b10c4b2205921a8a16eb031ffbeb69a1b4126e5cee7a4ebdddd761110b37970: Status 404 returned error can't find the container with id 2b10c4b2205921a8a16eb031ffbeb69a1b4126e5cee7a4ebdddd761110b37970 Jul 09 00:58:48.955436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.955414 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bg2z2" event={"ID":"883007c2-5de5-4310-ae50-fd129c1f53fe","Type":"ContainerStarted","Data":"63d6019fd4c1bbbe45151878896edd1b1012bd9454073de8c54912825f5d151e"} Jul 09 00:58:48.956523 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.956502 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" event={"ID":"e367f5d8-b32d-46ec-8202-72db8ca2323d","Type":"ContainerStarted","Data":"2b10c4b2205921a8a16eb031ffbeb69a1b4126e5cee7a4ebdddd761110b37970"} Jul 09 00:58:48.971617 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:48.971577 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bg2z2" podStartSLOduration=1.743095587 podStartE2EDuration="4.971563597s" totalImagesPullingTime="3.22846801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:58:48.970608914 +0000 UTC m=+193.248447564" watchObservedRunningTime="2026-07-09 00:58:48.971563597 +0000 UTC m=+193.249402250" Jul 09 00:58:52.975181 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:52.975148 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" event={"ID":"e367f5d8-b32d-46ec-8202-72db8ca2323d","Type":"ContainerStarted","Data":"ef4ef27c34e95c27b1fb0a7485a0ffc9260178fc2b08d517a8c64caa0432e723"} Jul 09 00:58:52.990457 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:52.990413 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x6zwv" podStartSLOduration=1.899602318 podStartE2EDuration="4.990399133s" totalImagesPullingTime="3.090796815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:58:52.989752118 +0000 UTC m=+197.267590769" watchObservedRunningTime="2026-07-09 00:58:52.990399133 +0000 UTC m=+197.268237784" Jul 09 00:58:53.664190 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:53.664154 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:53.764998 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:53.764965 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:58:55.917895 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:58:55.917864 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:11.374200 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:11.374164 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:11.474264 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:11.474232 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:12.189223 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.189189 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:12.392025 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.391984 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:12.507056 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.507024 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:12.574147 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.574118 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5"] Jul 09 00:59:12.633749 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.633719 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5"] Jul 09 00:59:12.633873 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.633811 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.636319 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.636299 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h7fwd\"" Jul 09 00:59:12.636453 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.636297 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 00:59:12.636453 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.636438 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 00:59:12.636545 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.636528 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 00:59:12.636591 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.636555 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 00:59:12.695133 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.695109 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:12.727245 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.727221 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/135d3237-3b4d-44a9-b438-831ad0d66d43-manifests\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.727317 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.727263 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/135d3237-3b4d-44a9-b438-831ad0d66d43-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.727317 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.727280 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/135d3237-3b4d-44a9-b438-831ad0d66d43-tmp\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.727393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.727359 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/135d3237-3b4d-44a9-b438-831ad0d66d43-webhook-cert\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.727393 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.727387 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmzm\" (UniqueName: \"kubernetes.io/projected/135d3237-3b4d-44a9-b438-831ad0d66d43-kube-api-access-hfmzm\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.797264 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.797191 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:12.828346 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.828320 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/135d3237-3b4d-44a9-b438-831ad0d66d43-webhook-cert\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.828458 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.828348 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmzm\" (UniqueName: \"kubernetes.io/projected/135d3237-3b4d-44a9-b438-831ad0d66d43-kube-api-access-hfmzm\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.828458 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.828389 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/135d3237-3b4d-44a9-b438-831ad0d66d43-manifests\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.828458 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.828414 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/135d3237-3b4d-44a9-b438-831ad0d66d43-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.828458 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.828434 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/135d3237-3b4d-44a9-b438-831ad0d66d43-tmp\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.828921 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.828892 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/135d3237-3b4d-44a9-b438-831ad0d66d43-manifests\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.830776 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.830754 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/135d3237-3b4d-44a9-b438-831ad0d66d43-tmp\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.830882 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.830828 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/135d3237-3b4d-44a9-b438-831ad0d66d43-webhook-cert\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.830882 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.830844 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/135d3237-3b4d-44a9-b438-831ad0d66d43-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.836046 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.836023 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmzm\" (UniqueName: \"kubernetes.io/projected/135d3237-3b4d-44a9-b438-831ad0d66d43-kube-api-access-hfmzm\") pod \"opendatahub-operator-controller-manager-85699f56b9-gxsq5\" (UID: \"135d3237-3b4d-44a9-b438-831ad0d66d43\") " pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.899049 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.899027 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:12.944044 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.944015 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:12.996008 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:12.995984 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:13.066115 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:13.066055 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:13.093131 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:13.093110 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5"] Jul 09 00:59:13.095165 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:59:13.095132 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135d3237_3b4d_44a9_b438_831ad0d66d43.slice/crio-de5fb55f6a36603a72cd2f5f6535ad958a2c6df1fb2d0e5cfc6188299bb34811 WatchSource:0}: Error finding container de5fb55f6a36603a72cd2f5f6535ad958a2c6df1fb2d0e5cfc6188299bb34811: Status 404 returned error can't find the container with id de5fb55f6a36603a72cd2f5f6535ad958a2c6df1fb2d0e5cfc6188299bb34811 Jul 09 00:59:14.051091 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:14.051048 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" event={"ID":"135d3237-3b4d-44a9-b438-831ad0d66d43","Type":"ContainerStarted","Data":"de5fb55f6a36603a72cd2f5f6535ad958a2c6df1fb2d0e5cfc6188299bb34811"} Jul 09 00:59:16.061829 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:16.061791 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" event={"ID":"135d3237-3b4d-44a9-b438-831ad0d66d43","Type":"ContainerStarted","Data":"ce2c719244b80b1bbd550839477fd46fdcd9a7c73c5fa2b997acbec453a4f834"} Jul 09 00:59:18.071753 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:18.071723 2543 generic.go:356] "Generic (PLEG): container finished" podID="135d3237-3b4d-44a9-b438-831ad0d66d43" containerID="ce2c719244b80b1bbd550839477fd46fdcd9a7c73c5fa2b997acbec453a4f834" exitCode=0 Jul 09 00:59:18.072090 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:18.071794 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" event={"ID":"135d3237-3b4d-44a9-b438-831ad0d66d43","Type":"ContainerDied","Data":"ce2c719244b80b1bbd550839477fd46fdcd9a7c73c5fa2b997acbec453a4f834"} Jul 09 00:59:19.077178 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:19.077147 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" event={"ID":"135d3237-3b4d-44a9-b438-831ad0d66d43","Type":"ContainerStarted","Data":"3027620f40d1fdb029575781b6eb536d4e8b002a9754a7c3f3aa478e4aa4eaf1"} Jul 09 00:59:19.077529 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:19.077351 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:19.098487 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:19.098444 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" podStartSLOduration=2.855150824 podStartE2EDuration="7.098430175s" totalImagesPullingTime="3.205566351s" totalInitContainerRuntime="1.037713s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:59:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:59:19.09642754 +0000 UTC m=+223.374266204" watchObservedRunningTime="2026-07-09 00:59:19.098430175 +0000 UTC m=+223.376268827" Jul 09 00:59:20.119556 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:20.119495 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:20.226416 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:20.226384 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:24.581146 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:24.581120 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:24.681450 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:24.681417 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:30.084427 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:30.084398 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85699f56b9-gxsq5" Jul 09 00:59:31.620241 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:31.620188 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:31.727126 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:31.727091 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:32.435953 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.435885 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:32.544627 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.544594 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:32.635146 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.635118 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:32.702756 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.702722 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:32.709121 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.709095 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t"] Jul 09 00:59:32.712137 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.712116 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.714775 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.714755 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 00:59:32.714885 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.714759 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 00:59:32.715354 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.715339 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 00:59:32.715804 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.715788 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 00:59:32.715896 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.715834 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tqfz8\"" Jul 09 00:59:32.726156 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.726135 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t"] Jul 09 00:59:32.767345 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.767322 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b596c63-6944-4722-ace5-8978ff269447-tmp\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.767435 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.767353 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b596c63-6944-4722-ace5-8978ff269447-tls-certs\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.767435 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.767404 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmm4\" (UniqueName: \"kubernetes.io/projected/5b596c63-6944-4722-ace5-8978ff269447-kube-api-access-gsmm4\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.868618 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.868590 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmm4\" (UniqueName: \"kubernetes.io/projected/5b596c63-6944-4722-ace5-8978ff269447-kube-api-access-gsmm4\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.868743 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.868638 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b596c63-6944-4722-ace5-8978ff269447-tmp\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.868743 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.868679 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b596c63-6944-4722-ace5-8978ff269447-tls-certs\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.870894 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.870875 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b596c63-6944-4722-ace5-8978ff269447-tmp\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.871134 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.871114 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b596c63-6944-4722-ace5-8978ff269447-tls-certs\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.876533 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.876507 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmm4\" (UniqueName: \"kubernetes.io/projected/5b596c63-6944-4722-ace5-8978ff269447-kube-api-access-gsmm4\") pod \"kube-auth-proxy-7556c75c9b-7tw9t\" (UID: \"5b596c63-6944-4722-ace5-8978ff269447\") " pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:32.919187 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:32.919158 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:33.022679 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:33.022605 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" Jul 09 00:59:33.156192 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:33.156170 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t"] Jul 09 00:59:33.158225 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:59:33.158192 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b596c63_6944_4722_ace5_8978ff269447.slice/crio-ecae9b7b2c1de2762cb6178e55be1596ac8dc64e4e91187527e35c77ae786e87 WatchSource:0}: Error finding container ecae9b7b2c1de2762cb6178e55be1596ac8dc64e4e91187527e35c77ae786e87: Status 404 returned error can't find the container with id ecae9b7b2c1de2762cb6178e55be1596ac8dc64e4e91187527e35c77ae786e87 Jul 09 00:59:33.199054 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:33.199032 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:33.463443 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:33.463381 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:33.735308 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:33.735211 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:33.802888 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:33.802859 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:34.142163 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:34.142085 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" event={"ID":"5b596c63-6944-4722-ace5-8978ff269447","Type":"ContainerStarted","Data":"ecae9b7b2c1de2762cb6178e55be1596ac8dc64e4e91187527e35c77ae786e87"} Jul 09 00:59:34.682405 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:34.682373 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:36.161356 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:36.161333 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 00:59:36.475277 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:36.475250 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:36.529778 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:36.529756 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:37.154816 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:37.154784 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" event={"ID":"5b596c63-6944-4722-ace5-8978ff269447","Type":"ContainerStarted","Data":"d88684ac1a3eb582fffd87209ae8dcb6320e21d5c1e85cf19c3acdcf77348655"} Jul 09 00:59:37.171438 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:37.171387 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7556c75c9b-7tw9t" podStartSLOduration=2.172507891 podStartE2EDuration="5.171371243s" totalImagesPullingTime="2.998863352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:59:37.170389646 +0000 UTC m=+241.448228298" watchObservedRunningTime="2026-07-09 00:59:37.171371243 +0000 UTC m=+241.449209901" Jul 09 00:59:39.317056 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.317023 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz"] Jul 09 00:59:39.320484 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.320468 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.326512 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.326488 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 00:59:39.326953 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.326748 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 00:59:39.327680 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.327663 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 00:59:39.328436 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.328423 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 00:59:39.328973 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.328954 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xndvk\"" Jul 09 00:59:39.348666 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.348630 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz"] Jul 09 00:59:39.419472 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.419448 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjrd\" (UniqueName: \"kubernetes.io/projected/036c5583-b903-4473-8c94-8dfb6fa73b91-kube-api-access-mkjrd\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.419574 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.419484 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.419574 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.419510 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.419574 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.419531 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c5583-b903-4473-8c94-8dfb6fa73b91-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.419726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.419590 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.419726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.419621 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.419726 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.419668 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.520529 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.520504 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjrd\" (UniqueName: \"kubernetes.io/projected/036c5583-b903-4473-8c94-8dfb6fa73b91-kube-api-access-mkjrd\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.520631 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.520541 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.520631 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.520565 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.520631 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.520586 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c5583-b903-4473-8c94-8dfb6fa73b91-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.520631 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.520610 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.520631 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.520627 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.520925 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.520665 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.521285 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.521264 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.522849 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.522829 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c5583-b903-4473-8c94-8dfb6fa73b91-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.522939 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.522881 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.523265 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.523244 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.523389 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.523369 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.528444 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.528418 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjrd\" (UniqueName: \"kubernetes.io/projected/036c5583-b903-4473-8c94-8dfb6fa73b91-kube-api-access-mkjrd\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.528524 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.528446 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036c5583-b903-4473-8c94-8dfb6fa73b91-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w7jdz\" (UID: \"036c5583-b903-4473-8c94-8dfb6fa73b91\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.629836 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.629763 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:39.755623 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:39.755600 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz"] Jul 09 00:59:39.757281 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:59:39.757249 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036c5583_b903_4473_8c94_8dfb6fa73b91.slice/crio-ab1769b2a1ba969f6641287b72cef453a4ac04ff0df91ecd3d2ea73026040cd7 WatchSource:0}: Error finding container ab1769b2a1ba969f6641287b72cef453a4ac04ff0df91ecd3d2ea73026040cd7: Status 404 returned error can't find the container with id ab1769b2a1ba969f6641287b72cef453a4ac04ff0df91ecd3d2ea73026040cd7 Jul 09 00:59:40.095022 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:40.094985 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:40.171093 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:40.171057 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" event={"ID":"036c5583-b903-4473-8c94-8dfb6fa73b91","Type":"ContainerStarted","Data":"ab1769b2a1ba969f6641287b72cef453a4ac04ff0df91ecd3d2ea73026040cd7"} Jul 09 00:59:40.182795 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:40.182761 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:40.316389 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:40.316355 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:40.417103 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:40.417033 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:40.762766 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:40.762735 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:41.980754 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:41.980717 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 00:59:41.981080 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:41.980805 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 00:59:42.131245 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.131206 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:42.189285 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.189224 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" event={"ID":"036c5583-b903-4473-8c94-8dfb6fa73b91","Type":"ContainerStarted","Data":"c037f182a26b8040a41f475082661893bd492fd36a3798e2b77ef6785cc18f4c"} Jul 09 00:59:42.189459 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.189404 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:42.216661 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.216597 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" podStartSLOduration=0.995619521 podStartE2EDuration="3.216578474s" totalImagesPullingTime="2.220958953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:59:42.215473891 +0000 UTC m=+246.493312544" watchObservedRunningTime="2026-07-09 00:59:42.216578474 +0000 UTC m=+246.494417126" Jul 09 00:59:42.217672 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.217630 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:42.508048 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.508008 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:42.584592 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.584552 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:42.601898 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.601867 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75764c764c-fn844"] Jul 09 00:59:42.605321 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.605306 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.607917 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.607885 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 00:59:42.608039 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.607913 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 00:59:42.608039 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.607929 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 00:59:42.608039 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.607917 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-p2cdx\"" Jul 09 00:59:42.613380 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.613354 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75764c764c-fn844"] Jul 09 00:59:42.648784 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.648762 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9k6\" (UniqueName: \"kubernetes.io/projected/cf26093c-ffc1-486d-866d-014facd70a06-kube-api-access-6v9k6\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.648915 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.648792 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf26093c-ffc1-486d-866d-014facd70a06-manager-config\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.648915 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.648840 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf26093c-ffc1-486d-866d-014facd70a06-metrics-cert\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.649020 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.648909 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf26093c-ffc1-486d-866d-014facd70a06-cert\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.749736 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.749695 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf26093c-ffc1-486d-866d-014facd70a06-cert\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.749918 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.749765 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9k6\" (UniqueName: \"kubernetes.io/projected/cf26093c-ffc1-486d-866d-014facd70a06-kube-api-access-6v9k6\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.749918 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.749797 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf26093c-ffc1-486d-866d-014facd70a06-manager-config\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.749918 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.749883 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf26093c-ffc1-486d-866d-014facd70a06-metrics-cert\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.750648 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.750617 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf26093c-ffc1-486d-866d-014facd70a06-manager-config\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.752843 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.752816 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf26093c-ffc1-486d-866d-014facd70a06-cert\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.753001 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.752860 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf26093c-ffc1-486d-866d-014facd70a06-metrics-cert\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.758374 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.758274 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9k6\" (UniqueName: \"kubernetes.io/projected/cf26093c-ffc1-486d-866d-014facd70a06-kube-api-access-6v9k6\") pod \"lws-controller-manager-75764c764c-fn844\" (UID: \"cf26093c-ffc1-486d-866d-014facd70a06\") " pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:42.915717 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:42.915682 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:43.045988 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:43.045964 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75764c764c-fn844"] Jul 09 00:59:43.048102 ip-10-0-134-128 kubenswrapper[2543]: W0709 00:59:43.047934 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf26093c_ffc1_486d_866d_014facd70a06.slice/crio-a2062374ee7242df11fc7108966fb209011effdc2fdc936fd0d7f321c81a9e45 WatchSource:0}: Error finding container a2062374ee7242df11fc7108966fb209011effdc2fdc936fd0d7f321c81a9e45: Status 404 returned error can't find the container with id a2062374ee7242df11fc7108966fb209011effdc2fdc936fd0d7f321c81a9e45 Jul 09 00:59:43.194213 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:43.194180 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" event={"ID":"cf26093c-ffc1-486d-866d-014facd70a06","Type":"ContainerStarted","Data":"a2062374ee7242df11fc7108966fb209011effdc2fdc936fd0d7f321c81a9e45"} Jul 09 00:59:43.195888 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:43.195862 2543 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-w7jdz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 00:59:43.196009 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:43.195913 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" podUID="036c5583-b903-4473-8c94-8dfb6fa73b91" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 00:59:43.424516 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:43.424444 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:43.528676 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:43.528630 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:43.589899 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:43.589859 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:45.206886 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:45.205739 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" event={"ID":"cf26093c-ffc1-486d-866d-014facd70a06","Type":"ContainerStarted","Data":"d455b554a7c092353d27e497a72d3ba1c919b09ab08930a0366f069d3c0c380f"} Jul 09 00:59:45.206886 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:45.206445 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 00:59:45.234576 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:45.234530 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" podStartSLOduration=2.010163774 podStartE2EDuration="3.234515216s" totalImagesPullingTime="1.224351442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 00:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 00:59:45.233160679 +0000 UTC m=+249.510999331" watchObservedRunningTime="2026-07-09 00:59:45.234515216 +0000 UTC m=+249.512353914" Jul 09 00:59:46.195101 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:46.195072 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w7jdz" Jul 09 00:59:55.115905 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:55.115869 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 00:59:57.216015 ip-10-0-134-128 kubenswrapper[2543]: I0709 00:59:57.215985 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75764c764c-fn844" Jul 09 01:00:01.645903 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:01.645817 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:01.699604 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:01.699571 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:03.544201 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:03.544167 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:03.609363 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:03.609338 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:20.628315 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:20.628282 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:33.051897 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:33.051859 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:33.149329 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:33.149297 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:33.249572 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:33.249545 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:00:36.134695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:00:36.134669 2543 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 01:01:02.067663 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:02.067612 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:02.663400 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:02.663365 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:03.233330 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:03.233292 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:07.472574 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:07.472534 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:09.583993 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:09.583956 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:09.684128 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:09.684093 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:10.471680 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:10.471627 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:11.296328 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:11.296292 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:13.467970 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:13.467929 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:13.923999 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:13.923934 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:17.178842 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:17.178763 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:19.069481 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:19.069447 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:19.175088 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:19.175062 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:19.474698 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:19.474671 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:19.572373 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:19.572338 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:22.907733 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:22.907695 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:23.008627 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:23.008597 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:26.278180 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:26.278145 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:26.376083 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:26.376053 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:27.013957 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:27.013927 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:27.230876 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:27.230840 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:30.214553 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:30.214460 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:30.312018 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:30.311965 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:38.273533 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:38.273495 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:38.370605 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:38.370577 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:38.899468 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:38.899428 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:38.994730 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:38.994703 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:42.271752 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:42.271719 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:42.373231 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:42.373194 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:44.656199 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:44.656172 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:44.708273 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:44.708245 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:46.078706 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:46.078662 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:46.129615 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:46.129591 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:50.059858 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:50.059823 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:50.112526 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:50.112493 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:57.846877 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:57.846840 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:57.908589 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:57.908564 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:01:57.962524 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:01:57.962495 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:19.121393 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:19.121364 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:19.431762 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:19.431686 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:32.678412 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:32.678363 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:32.786443 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:32.786410 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:32.840520 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:32.840491 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:33.512987 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:33.512957 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:33.612759 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:33.612730 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:51.429712 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:51.429681 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:51.541727 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:51.541690 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:02:51.976459 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:02:51.976428 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:04.437993 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:04.437918 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:04.538059 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:04.538026 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:11.279987 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:11.279954 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:11.379579 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:11.379526 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:11.431321 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:11.431291 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:11.778414 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:11.778386 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:14.713519 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:14.713484 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:14.813813 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:14.813777 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:44.165058 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:44.165028 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:47.024783 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:47.024743 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:47.434214 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:47.434142 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:47.610429 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:47.610395 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:47.838514 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:47.838484 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:47.945216 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:47.945185 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:58.500992 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:58.500951 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:03:58.553266 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:03:58.553237 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:09.210766 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:09.210728 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:15.247402 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:15.247370 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:25.492089 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:25.492049 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:25.546631 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:25.546601 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:30.252675 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:30.252588 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:41.282751 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:41.282718 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:44.753862 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:44.753830 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:04:55.356528 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:04:55.356491 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:05:01.154723 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:05:01.154688 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:05:43.131195 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:05:43.131161 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:06:11.037635 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:06:11.037601 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:06:11.132537 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:06:11.132499 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:06:12.047855 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:06:12.047802 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:06:12.146179 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:06:12.146147 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:06:56.539463 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:06:56.539406 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:06:56.662904 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:06:56.662877 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:07:23.072088 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:07:23.072056 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:07:24.405740 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:07:24.405704 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:08.267680 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:08.267627 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:08.370081 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:08.370055 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:08.471515 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:08.471486 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:13.959613 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:13.959581 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:14.061891 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:14.061859 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:14.160476 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:14.160442 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:14.300813 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:14.300729 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:19.929438 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:19.929405 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:20.026634 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:20.026603 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:08:31.822409 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:08:31.822374 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:09:18.752872 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:09:18.752765 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:11:44.354077 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:11:44.354038 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:13:58.284021 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:13:58.283991 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:13:58.370610 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:13:58.370583 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:15:00.142729 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.142635 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb"] Jul 09 01:15:00.146227 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.146207 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" Jul 09 01:15:00.148402 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.148383 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:15:00.149485 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.149469 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:15:00.149570 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.149519 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:15:00.152194 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.152175 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb"] Jul 09 01:15:00.264786 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.264762 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbkm\" (UniqueName: \"kubernetes.io/projected/aa9c05cd-815b-4c24-8d53-a8c38f0678d5-kube-api-access-8hbkm\") pod \"maas-api-key-cleanup-29725995-jqpvb\" (UID: \"aa9c05cd-815b-4c24-8d53-a8c38f0678d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" Jul 09 01:15:00.366046 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.366009 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbkm\" (UniqueName: \"kubernetes.io/projected/aa9c05cd-815b-4c24-8d53-a8c38f0678d5-kube-api-access-8hbkm\") pod \"maas-api-key-cleanup-29725995-jqpvb\" (UID: \"aa9c05cd-815b-4c24-8d53-a8c38f0678d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" Jul 09 01:15:00.373510 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.373485 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbkm\" (UniqueName: \"kubernetes.io/projected/aa9c05cd-815b-4c24-8d53-a8c38f0678d5-kube-api-access-8hbkm\") pod \"maas-api-key-cleanup-29725995-jqpvb\" (UID: \"aa9c05cd-815b-4c24-8d53-a8c38f0678d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" Jul 09 01:15:00.457528 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.457506 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" Jul 09 01:15:00.580031 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.580001 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb"] Jul 09 01:15:00.583104 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:15:00.583077 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9c05cd_815b_4c24_8d53_a8c38f0678d5.slice/crio-47905f331f6e4eaf110bcb378fecfec8a681c5a6845e1b35601649bb7633a1f5 WatchSource:0}: Error finding container 47905f331f6e4eaf110bcb378fecfec8a681c5a6845e1b35601649bb7633a1f5: Status 404 returned error can't find the container with id 47905f331f6e4eaf110bcb378fecfec8a681c5a6845e1b35601649bb7633a1f5 Jul 09 01:15:00.584818 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.584799 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 01:15:00.919197 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:00.919117 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" event={"ID":"aa9c05cd-815b-4c24-8d53-a8c38f0678d5","Type":"ContainerStarted","Data":"47905f331f6e4eaf110bcb378fecfec8a681c5a6845e1b35601649bb7633a1f5"} Jul 09 01:15:09.961255 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:09.961218 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" event={"ID":"aa9c05cd-815b-4c24-8d53-a8c38f0678d5","Type":"ContainerStarted","Data":"8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59"} Jul 09 01:15:09.975757 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:15:09.975709 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" podStartSLOduration=1.419706018 podStartE2EDuration="9.975693998s" totalImagesPullingTime="8.55598798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:15:09.974813369 +0000 UTC m=+1174.252652021" watchObservedRunningTime="2026-07-09 01:15:09.975693998 +0000 UTC m=+1174.253532648" Jul 09 01:16:00.010569 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:00.010537 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb"] Jul 09 01:16:00.011134 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:00.010753 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" podUID="aa9c05cd-815b-4c24-8d53-a8c38f0678d5" containerName="cleanup" containerID="cri-o://8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59" gracePeriod=30 Jul 09 01:16:30.189617 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.189593 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" Jul 09 01:16:30.285111 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.285031 2543 generic.go:356] "Generic (PLEG): container finished" podID="aa9c05cd-815b-4c24-8d53-a8c38f0678d5" containerID="8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59" exitCode=137 Jul 09 01:16:30.285111 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.285078 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" Jul 09 01:16:30.285290 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.285112 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" event={"ID":"aa9c05cd-815b-4c24-8d53-a8c38f0678d5","Type":"ContainerDied","Data":"8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59"} Jul 09 01:16:30.285290 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.285183 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb" event={"ID":"aa9c05cd-815b-4c24-8d53-a8c38f0678d5","Type":"ContainerDied","Data":"47905f331f6e4eaf110bcb378fecfec8a681c5a6845e1b35601649bb7633a1f5"} Jul 09 01:16:30.285290 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.285212 2543 scope.go:117] "RemoveContainer" containerID="8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59" Jul 09 01:16:30.292867 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.292850 2543 scope.go:117] "RemoveContainer" containerID="8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59" Jul 09 01:16:30.293137 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:16:30.293114 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59\": container with ID starting with 8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59 not found: ID does not exist" containerID="8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59" Jul 09 01:16:30.293237 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.293142 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59"} err="failed to get container status \"8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59\": rpc error: code = NotFound desc = could not find container \"8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59\": container with ID starting with 8d767cfd4825f2e6777a140f7d4d5d7882967a754586864f98db13d1ef480b59 not found: ID does not exist" Jul 09 01:16:30.316010 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.315988 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hbkm\" (UniqueName: \"kubernetes.io/projected/aa9c05cd-815b-4c24-8d53-a8c38f0678d5-kube-api-access-8hbkm\") pod \"aa9c05cd-815b-4c24-8d53-a8c38f0678d5\" (UID: \"aa9c05cd-815b-4c24-8d53-a8c38f0678d5\") " Jul 09 01:16:30.317977 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.317958 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9c05cd-815b-4c24-8d53-a8c38f0678d5-kube-api-access-8hbkm" (OuterVolumeSpecName: "kube-api-access-8hbkm") pod "aa9c05cd-815b-4c24-8d53-a8c38f0678d5" (UID: "aa9c05cd-815b-4c24-8d53-a8c38f0678d5"). InnerVolumeSpecName "kube-api-access-8hbkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:16:30.417254 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.417221 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbkm\" (UniqueName: \"kubernetes.io/projected/aa9c05cd-815b-4c24-8d53-a8c38f0678d5-kube-api-access-8hbkm\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:16:30.605410 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.605378 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb"] Jul 09 01:16:30.609069 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.609044 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725995-jqpvb"] Jul 09 01:16:30.928369 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:30.928292 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9c05cd-815b-4c24-8d53-a8c38f0678d5" path="/var/lib/kubelet/pods/aa9c05cd-815b-4c24-8d53-a8c38f0678d5/volumes" Jul 09 01:16:53.358402 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:16:53.358363 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:05.795554 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:05.795520 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:17.450605 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:17.450566 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:25.707538 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:25.707507 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:30.812343 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:30.812264 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:33.610826 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:33.610757 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:33.711927 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:33.711893 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:34.053283 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:34.053247 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:39.529561 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:39.529521 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:39.633498 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:39.633460 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:39.729313 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:39.729268 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:39.830301 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:39.830224 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:40.822448 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:40.822416 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:40.922352 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:40.922316 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:41.031427 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.031395 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq"] Jul 09 01:19:41.031939 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.031919 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9c05cd-815b-4c24-8d53-a8c38f0678d5" containerName="cleanup" Jul 09 01:19:41.031997 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.031943 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9c05cd-815b-4c24-8d53-a8c38f0678d5" containerName="cleanup" Jul 09 01:19:41.032052 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.032040 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9c05cd-815b-4c24-8d53-a8c38f0678d5" containerName="cleanup" Jul 09 01:19:41.035273 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.035254 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.037514 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.037478 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a486953c-serving-cert\"" Jul 09 01:19:41.037627 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.037483 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:19:41.037711 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.037667 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:19:41.038400 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.038384 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:19:41.043691 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.043635 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq"] Jul 09 01:19:41.067397 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.067370 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:41.113691 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.113619 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed545d0-227f-460c-9f2c-a35c8ec986c2-openshift-service-ca\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.113799 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.113702 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed545d0-227f-460c-9f2c-a35c8ec986c2-maas-api-tls\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.113799 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.113722 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv587\" (UniqueName: \"kubernetes.io/projected/9ed545d0-227f-460c-9f2c-a35c8ec986c2-kube-api-access-zv587\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.214113 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.214075 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed545d0-227f-460c-9f2c-a35c8ec986c2-openshift-service-ca\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.214284 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.214157 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed545d0-227f-460c-9f2c-a35c8ec986c2-maas-api-tls\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.214284 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.214178 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv587\" (UniqueName: \"kubernetes.io/projected/9ed545d0-227f-460c-9f2c-a35c8ec986c2-kube-api-access-zv587\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.214737 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.214712 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed545d0-227f-460c-9f2c-a35c8ec986c2-openshift-service-ca\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.216551 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.216532 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed545d0-227f-460c-9f2c-a35c8ec986c2-maas-api-tls\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.221449 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.221427 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv587\" (UniqueName: \"kubernetes.io/projected/9ed545d0-227f-460c-9f2c-a35c8ec986c2-kube-api-access-zv587\") pod \"maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.238364 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.238329 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:41.346953 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.346915 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:41.469999 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.469968 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:41.479484 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.479454 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq"] Jul 09 01:19:41.482470 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:19:41.482443 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed545d0_227f_460c_9f2c_a35c8ec986c2.slice/crio-f6f8646eacaf25895962ab32a85bb52490110a94a50d07b3d6cf0db4ee8e9fce WatchSource:0}: Error finding container f6f8646eacaf25895962ab32a85bb52490110a94a50d07b3d6cf0db4ee8e9fce: Status 404 returned error can't find the container with id f6f8646eacaf25895962ab32a85bb52490110a94a50d07b3d6cf0db4ee8e9fce Jul 09 01:19:41.571546 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:41.571494 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:42.072168 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:42.072136 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" event={"ID":"9ed545d0-227f-460c-9f2c-a35c8ec986c2","Type":"ContainerStarted","Data":"f6f8646eacaf25895962ab32a85bb52490110a94a50d07b3d6cf0db4ee8e9fce"} Jul 09 01:19:44.082203 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:44.082168 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" event={"ID":"9ed545d0-227f-460c-9f2c-a35c8ec986c2","Type":"ContainerStarted","Data":"60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264"} Jul 09 01:19:44.082584 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:44.082277 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:44.100246 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:44.100198 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" podStartSLOduration=1.063484904 podStartE2EDuration="3.100183282s" totalImagesPullingTime="2.036698378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:19:44.097677105 +0000 UTC m=+1448.375515756" watchObservedRunningTime="2026-07-09 01:19:44.100183282 +0000 UTC m=+1448.378021932" Jul 09 01:19:44.750933 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:44.750891 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:45.919248 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:45.919209 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:46.145240 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.145207 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:46.210819 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.210776 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:46.331984 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.331954 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq"] Jul 09 01:19:46.332192 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.332164 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" podUID="9ed545d0-227f-460c-9f2c-a35c8ec986c2" containerName="maas-api" containerID="cri-o://60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264" gracePeriod=30 Jul 09 01:19:46.354274 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.354249 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:46.573783 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.573758 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:46.660628 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.660603 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv587\" (UniqueName: \"kubernetes.io/projected/9ed545d0-227f-460c-9f2c-a35c8ec986c2-kube-api-access-zv587\") pod \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " Jul 09 01:19:46.660786 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.660636 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed545d0-227f-460c-9f2c-a35c8ec986c2-openshift-service-ca\") pod \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " Jul 09 01:19:46.660786 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.660701 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed545d0-227f-460c-9f2c-a35c8ec986c2-maas-api-tls\") pod \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\" (UID: \"9ed545d0-227f-460c-9f2c-a35c8ec986c2\") " Jul 09 01:19:46.661044 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.661019 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed545d0-227f-460c-9f2c-a35c8ec986c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ed545d0-227f-460c-9f2c-a35c8ec986c2" (UID: "9ed545d0-227f-460c-9f2c-a35c8ec986c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:19:46.662514 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.662494 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed545d0-227f-460c-9f2c-a35c8ec986c2-kube-api-access-zv587" (OuterVolumeSpecName: "kube-api-access-zv587") pod "9ed545d0-227f-460c-9f2c-a35c8ec986c2" (UID: "9ed545d0-227f-460c-9f2c-a35c8ec986c2"). InnerVolumeSpecName "kube-api-access-zv587". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:19:46.662638 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.662607 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed545d0-227f-460c-9f2c-a35c8ec986c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ed545d0-227f-460c-9f2c-a35c8ec986c2" (UID: "9ed545d0-227f-460c-9f2c-a35c8ec986c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:19:46.761414 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.761342 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed545d0-227f-460c-9f2c-a35c8ec986c2-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:19:46.761414 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.761370 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed545d0-227f-460c-9f2c-a35c8ec986c2-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:19:46.761414 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:46.761380 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv587\" (UniqueName: \"kubernetes.io/projected/9ed545d0-227f-460c-9f2c-a35c8ec986c2-kube-api-access-zv587\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:19:47.096612 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.096504 2543 generic.go:356] "Generic (PLEG): container finished" podID="9ed545d0-227f-460c-9f2c-a35c8ec986c2" containerID="60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264" exitCode=0 Jul 09 01:19:47.096612 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.096574 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" Jul 09 01:19:47.096612 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.096594 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" event={"ID":"9ed545d0-227f-460c-9f2c-a35c8ec986c2","Type":"ContainerDied","Data":"60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264"} Jul 09 01:19:47.097172 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.096674 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq" event={"ID":"9ed545d0-227f-460c-9f2c-a35c8ec986c2","Type":"ContainerDied","Data":"f6f8646eacaf25895962ab32a85bb52490110a94a50d07b3d6cf0db4ee8e9fce"} Jul 09 01:19:47.097172 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.096709 2543 scope.go:117] "RemoveContainer" containerID="60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264" Jul 09 01:19:47.104844 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.104829 2543 scope.go:117] "RemoveContainer" containerID="60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264" Jul 09 01:19:47.105082 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:19:47.105066 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264\": container with ID starting with 60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264 not found: ID does not exist" containerID="60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264" Jul 09 01:19:47.105134 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.105091 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264"} err="failed to get container status \"60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264\": rpc error: code = NotFound desc = could not find container \"60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264\": container with ID starting with 60c70b315147ab70bef2ab78abfed057d68dd00e7d8e41e85475c7a269f53264 not found: ID does not exist" Jul 09 01:19:47.112758 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.112735 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq"] Jul 09 01:19:47.116146 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:47.116127 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a486953c-594bf9d55d-nfzdq"] Jul 09 01:19:48.091948 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:48.091920 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:48.927722 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:48.927692 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed545d0-227f-460c-9f2c-a35c8ec986c2" path="/var/lib/kubelet/pods/9ed545d0-227f-460c-9f2c-a35c8ec986c2/volumes" Jul 09 01:19:51.926444 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:51.926405 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:51.984982 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:51.984948 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:52.113411 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:52.113376 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:52.212982 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:52.212951 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:53.219496 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:53.219457 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:53.319247 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:53.319220 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:53.439083 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:53.439048 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:53.977084 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:53.977042 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:54.078345 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:54.078313 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:54.179960 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:54.179926 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:55.684655 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:55.684623 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:58.335811 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:58.335775 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:19:58.668489 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:19:58.668414 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:04.403836 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:04.403802 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:04.506731 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:04.506691 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:07.294508 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:07.294473 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:07.763401 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:07.763365 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:10.423991 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:10.423956 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:10.487034 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:10.486998 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:10.627637 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:10.627606 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:10.728764 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:10.728729 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:11.716374 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.716326 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:11.784544 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.784505 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:11.789459 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.789434 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g"] Jul 09 01:20:11.789869 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.789855 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed545d0-227f-460c-9f2c-a35c8ec986c2" containerName="maas-api" Jul 09 01:20:11.789960 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.789870 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed545d0-227f-460c-9f2c-a35c8ec986c2" containerName="maas-api" Jul 09 01:20:11.789960 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.789931 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed545d0-227f-460c-9f2c-a35c8ec986c2" containerName="maas-api" Jul 09 01:20:11.792831 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.792814 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.795352 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.795333 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-488986e9-serving-cert\"" Jul 09 01:20:11.795451 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.795429 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:20:11.795862 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.795846 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:20:11.796711 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.796691 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:20:11.802544 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.802523 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g"] Jul 09 01:20:11.856077 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.856051 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:11.866111 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.866092 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.866196 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.866142 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ce3ab-5637-476f-9487-e93d760e69a0-openshift-service-ca\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.866196 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.866172 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drsvz\" (UniqueName: \"kubernetes.io/projected/8b9ce3ab-5637-476f-9487-e93d760e69a0-kube-api-access-drsvz\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.967114 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.967036 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ce3ab-5637-476f-9487-e93d760e69a0-openshift-service-ca\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.967114 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.967095 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drsvz\" (UniqueName: \"kubernetes.io/projected/8b9ce3ab-5637-476f-9487-e93d760e69a0-kube-api-access-drsvz\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.967270 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.967181 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.967311 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:20:11.967288 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-serving-cert: secret "maas-api-e2e-derive-488986e9-serving-cert" not found Jul 09 01:20:11.967385 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:20:11.967372 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls podName:8b9ce3ab-5637-476f-9487-e93d760e69a0 nodeName:}" failed. No retries permitted until 2026-07-09 01:20:12.467350417 +0000 UTC m=+1476.745189060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls") pod "maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" (UID: "8b9ce3ab-5637-476f-9487-e93d760e69a0") : secret "maas-api-e2e-derive-488986e9-serving-cert" not found Jul 09 01:20:11.967750 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.967735 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ce3ab-5637-476f-9487-e93d760e69a0-openshift-service-ca\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:11.977999 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:11.977976 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drsvz\" (UniqueName: \"kubernetes.io/projected/8b9ce3ab-5637-476f-9487-e93d760e69a0-kube-api-access-drsvz\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:12.171044 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.171012 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:12.360190 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.360103 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:12.460977 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.460947 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:12.471075 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.471047 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:12.473316 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.473297 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls\") pod \"maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:12.703837 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.703809 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:12.834287 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.834262 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g"] Jul 09 01:20:12.836413 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:20:12.836387 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9ce3ab_5637_476f_9487_e93d760e69a0.slice/crio-5a17a1a2e959e5741acdc4a5771d1d2f45b6650da59561c8b0d7df07aabf560c WatchSource:0}: Error finding container 5a17a1a2e959e5741acdc4a5771d1d2f45b6650da59561c8b0d7df07aabf560c: Status 404 returned error can't find the container with id 5a17a1a2e959e5741acdc4a5771d1d2f45b6650da59561c8b0d7df07aabf560c Jul 09 01:20:12.838161 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:12.838141 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 01:20:13.202842 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:13.202818 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" event={"ID":"8b9ce3ab-5637-476f-9487-e93d760e69a0","Type":"ContainerStarted","Data":"5a17a1a2e959e5741acdc4a5771d1d2f45b6650da59561c8b0d7df07aabf560c"} Jul 09 01:20:13.973421 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:13.973389 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:14.209458 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:14.209369 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" event={"ID":"8b9ce3ab-5637-476f-9487-e93d760e69a0","Type":"ContainerStarted","Data":"cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9"} Jul 09 01:20:14.209608 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:14.209480 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:14.225661 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:14.225594 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" podStartSLOduration=2.909710714 podStartE2EDuration="3.225579332s" totalImagesPullingTime="315.868618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:20:14.223743446 +0000 UTC m=+1478.501582124" watchObservedRunningTime="2026-07-09 01:20:14.225579332 +0000 UTC m=+1478.503417983" Jul 09 01:20:16.213078 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.213045 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:16.388035 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.387991 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:16.528220 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.528135 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:16.576118 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.576091 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g"] Jul 09 01:20:16.576292 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.576259 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" podUID="8b9ce3ab-5637-476f-9487-e93d760e69a0" containerName="maas-api" containerID="cri-o://cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9" gracePeriod=30 Jul 09 01:20:16.595413 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.595388 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:16.814581 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.814559 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:16.908017 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.907986 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ce3ab-5637-476f-9487-e93d760e69a0-openshift-service-ca\") pod \"8b9ce3ab-5637-476f-9487-e93d760e69a0\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " Jul 09 01:20:16.908189 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.908069 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drsvz\" (UniqueName: \"kubernetes.io/projected/8b9ce3ab-5637-476f-9487-e93d760e69a0-kube-api-access-drsvz\") pod \"8b9ce3ab-5637-476f-9487-e93d760e69a0\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " Jul 09 01:20:16.908189 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.908120 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls\") pod \"8b9ce3ab-5637-476f-9487-e93d760e69a0\" (UID: \"8b9ce3ab-5637-476f-9487-e93d760e69a0\") " Jul 09 01:20:16.908385 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.908365 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ce3ab-5637-476f-9487-e93d760e69a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b9ce3ab-5637-476f-9487-e93d760e69a0" (UID: "8b9ce3ab-5637-476f-9487-e93d760e69a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:20:16.910115 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.910091 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b9ce3ab-5637-476f-9487-e93d760e69a0" (UID: "8b9ce3ab-5637-476f-9487-e93d760e69a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:20:16.910218 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:16.910162 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9ce3ab-5637-476f-9487-e93d760e69a0-kube-api-access-drsvz" (OuterVolumeSpecName: "kube-api-access-drsvz") pod "8b9ce3ab-5637-476f-9487-e93d760e69a0" (UID: "8b9ce3ab-5637-476f-9487-e93d760e69a0"). InnerVolumeSpecName "kube-api-access-drsvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:20:17.008948 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.008922 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9ce3ab-5637-476f-9487-e93d760e69a0-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:20:17.008948 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.008946 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ce3ab-5637-476f-9487-e93d760e69a0-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:20:17.009081 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.008955 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drsvz\" (UniqueName: \"kubernetes.io/projected/8b9ce3ab-5637-476f-9487-e93d760e69a0-kube-api-access-drsvz\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:20:17.224709 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.224679 2543 generic.go:356] "Generic (PLEG): container finished" podID="8b9ce3ab-5637-476f-9487-e93d760e69a0" containerID="cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9" exitCode=0 Jul 09 01:20:17.225125 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.224746 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" event={"ID":"8b9ce3ab-5637-476f-9487-e93d760e69a0","Type":"ContainerDied","Data":"cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9"} Jul 09 01:20:17.225125 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.224783 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" event={"ID":"8b9ce3ab-5637-476f-9487-e93d760e69a0","Type":"ContainerDied","Data":"5a17a1a2e959e5741acdc4a5771d1d2f45b6650da59561c8b0d7df07aabf560c"} Jul 09 01:20:17.225125 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.224804 2543 scope.go:117] "RemoveContainer" containerID="cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9" Jul 09 01:20:17.225125 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.224744 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g" Jul 09 01:20:17.234331 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.234314 2543 scope.go:117] "RemoveContainer" containerID="cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9" Jul 09 01:20:17.235056 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:20:17.235034 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9\": container with ID starting with cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9 not found: ID does not exist" containerID="cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9" Jul 09 01:20:17.235184 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.235064 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9"} err="failed to get container status \"cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9\": rpc error: code = NotFound desc = could not find container \"cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9\": container with ID starting with cd9444bb5730bcf24ac258b5a0d54862fbd3e3d4afcb9a0bf5d36d1a2282c1b9 not found: ID does not exist" Jul 09 01:20:17.239927 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.239906 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g"] Jul 09 01:20:17.243867 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.243849 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-488986e9-79d7cc8cbc-7t42g"] Jul 09 01:20:17.754808 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:17.754779 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:18.928319 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:18.928292 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9ce3ab-5637-476f-9487-e93d760e69a0" path="/var/lib/kubelet/pods/8b9ce3ab-5637-476f-9487-e93d760e69a0/volumes" Jul 09 01:20:22.044915 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:22.044881 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:22.150890 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:22.150856 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:22.516685 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:22.516656 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:22.582133 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:22.582102 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:22.650366 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:22.650334 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:24.114223 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:24.114184 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:29.788938 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:29.788901 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:29.850268 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:29.850239 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:29.929431 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:29.929396 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:30.044043 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:30.043955 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:30.142594 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:30.142550 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:30.544062 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:30.544021 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:33.906674 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:33.906627 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:33.965067 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:33.965036 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:20:56.505343 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:20:56.505313 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:02.247572 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:02.247492 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:02.355968 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:02.355928 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:02.448904 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:02.448854 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:18.333614 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:18.333577 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:23.917356 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:23.917325 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:23.981090 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:23.981060 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:24.055492 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:24.055459 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:24.156889 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:24.156856 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:34.627081 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:34.627045 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:35.422390 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:35.422358 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:41.078996 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:41.078961 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:41.186336 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:41.186302 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:41.279847 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:41.279801 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:41.379190 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:41.379110 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:42.163333 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:42.163244 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:42.254420 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:42.254381 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:42.354294 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:42.354257 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:43.983027 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:43.982994 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:49.588458 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:49.588421 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:49.688290 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:49.688254 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:50.285352 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:50.285321 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:56.019068 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:56.019032 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:57.351304 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:57.351266 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:59.364927 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.364888 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:59.455044 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.455014 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp"] Jul 09 01:21:59.455338 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.455324 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ce3ab-5637-476f-9487-e93d760e69a0" containerName="maas-api" Jul 09 01:21:59.455384 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.455339 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ce3ab-5637-476f-9487-e93d760e69a0" containerName="maas-api" Jul 09 01:21:59.455418 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.455393 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ce3ab-5637-476f-9487-e93d760e69a0" containerName="maas-api" Jul 09 01:21:59.457475 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.457443 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:59.457909 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.457892 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.460325 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.460306 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:21:59.460669 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.460632 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:21:59.461591 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.461577 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b5edbb9-serving-cert\"" Jul 09 01:21:59.461671 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.461636 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:21:59.470973 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.470948 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp"] Jul 09 01:21:59.523528 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.523501 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:59.590955 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.590931 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48d6b00-8c61-4e36-b7e5-9098d270452f-openshift-service-ca\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.591097 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.590971 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw7gt\" (UniqueName: \"kubernetes.io/projected/c48d6b00-8c61-4e36-b7e5-9098d270452f-kube-api-access-rw7gt\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.591097 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.590998 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48d6b00-8c61-4e36-b7e5-9098d270452f-maas-api-tls\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.623777 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.623715 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:21:59.691448 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.691412 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw7gt\" (UniqueName: \"kubernetes.io/projected/c48d6b00-8c61-4e36-b7e5-9098d270452f-kube-api-access-rw7gt\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.691585 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.691452 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48d6b00-8c61-4e36-b7e5-9098d270452f-maas-api-tls\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.691585 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.691497 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48d6b00-8c61-4e36-b7e5-9098d270452f-openshift-service-ca\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.692192 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.692170 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48d6b00-8c61-4e36-b7e5-9098d270452f-openshift-service-ca\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.693882 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.693859 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48d6b00-8c61-4e36-b7e5-9098d270452f-maas-api-tls\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.699705 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.699684 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw7gt\" (UniqueName: \"kubernetes.io/projected/c48d6b00-8c61-4e36-b7e5-9098d270452f-kube-api-access-rw7gt\") pod \"maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.768481 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.768441 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:21:59.894179 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.894155 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp"] Jul 09 01:21:59.896104 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:21:59.896070 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48d6b00_8c61_4e36_b7e5_9098d270452f.slice/crio-3bc819f8fe5c565981853167bb83579e581efcd9a0ef5efa3019aaa8433961f7 WatchSource:0}: Error finding container 3bc819f8fe5c565981853167bb83579e581efcd9a0ef5efa3019aaa8433961f7: Status 404 returned error can't find the container with id 3bc819f8fe5c565981853167bb83579e581efcd9a0ef5efa3019aaa8433961f7 Jul 09 01:21:59.911539 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:21:59.911517 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:00.012358 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:00.012316 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:00.664514 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:00.664487 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" event={"ID":"c48d6b00-8c61-4e36-b7e5-9098d270452f","Type":"ContainerStarted","Data":"3bc819f8fe5c565981853167bb83579e581efcd9a0ef5efa3019aaa8433961f7"} Jul 09 01:22:01.670359 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:01.670326 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" event={"ID":"c48d6b00-8c61-4e36-b7e5-9098d270452f","Type":"ContainerStarted","Data":"40bda183adc5c2a02f3082177606613fea9f3ea88dfde18065a309a02ea7858f"} Jul 09 01:22:01.670753 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:01.670434 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:22:01.688171 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:01.688116 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" podStartSLOduration=1.939172214 podStartE2EDuration="2.68810136s" totalImagesPullingTime="748.929146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:22:01.685890719 +0000 UTC m=+1585.963729371" watchObservedRunningTime="2026-07-09 01:22:01.68810136 +0000 UTC m=+1585.965940011" Jul 09 01:22:03.886653 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:03.886615 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:04.575559 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:04.575518 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:04.675559 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:04.675516 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:05.026102 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:05.026069 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:06.372259 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:06.372230 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:06.472305 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:06.472271 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:07.680388 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:07.680360 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:22:10.653198 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:10.653162 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:10.753012 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:10.752978 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:11.101382 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.101344 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:11.483683 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.483537 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp"] Jul 09 01:22:11.483877 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.483794 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" podUID="c48d6b00-8c61-4e36-b7e5-9098d270452f" containerName="maas-api" containerID="cri-o://40bda183adc5c2a02f3082177606613fea9f3ea88dfde18065a309a02ea7858f" gracePeriod=30 Jul 09 01:22:11.511673 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.511624 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:11.562400 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.562355 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:11.712912 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.712885 2543 generic.go:356] "Generic (PLEG): container finished" podID="c48d6b00-8c61-4e36-b7e5-9098d270452f" containerID="40bda183adc5c2a02f3082177606613fea9f3ea88dfde18065a309a02ea7858f" exitCode=0 Jul 09 01:22:11.713266 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.712963 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" event={"ID":"c48d6b00-8c61-4e36-b7e5-9098d270452f","Type":"ContainerDied","Data":"40bda183adc5c2a02f3082177606613fea9f3ea88dfde18065a309a02ea7858f"} Jul 09 01:22:11.713266 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.713059 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" event={"ID":"c48d6b00-8c61-4e36-b7e5-9098d270452f","Type":"ContainerDied","Data":"3bc819f8fe5c565981853167bb83579e581efcd9a0ef5efa3019aaa8433961f7"} Jul 09 01:22:11.713266 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.713085 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc819f8fe5c565981853167bb83579e581efcd9a0ef5efa3019aaa8433961f7" Jul 09 01:22:11.725611 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.725590 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:22:11.891669 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.891560 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw7gt\" (UniqueName: \"kubernetes.io/projected/c48d6b00-8c61-4e36-b7e5-9098d270452f-kube-api-access-rw7gt\") pod \"c48d6b00-8c61-4e36-b7e5-9098d270452f\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " Jul 09 01:22:11.891669 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.891609 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48d6b00-8c61-4e36-b7e5-9098d270452f-maas-api-tls\") pod \"c48d6b00-8c61-4e36-b7e5-9098d270452f\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " Jul 09 01:22:11.891899 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.891689 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48d6b00-8c61-4e36-b7e5-9098d270452f-openshift-service-ca\") pod \"c48d6b00-8c61-4e36-b7e5-9098d270452f\" (UID: \"c48d6b00-8c61-4e36-b7e5-9098d270452f\") " Jul 09 01:22:11.892049 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.892023 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48d6b00-8c61-4e36-b7e5-9098d270452f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c48d6b00-8c61-4e36-b7e5-9098d270452f" (UID: "c48d6b00-8c61-4e36-b7e5-9098d270452f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:22:11.893733 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.893712 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48d6b00-8c61-4e36-b7e5-9098d270452f-kube-api-access-rw7gt" (OuterVolumeSpecName: "kube-api-access-rw7gt") pod "c48d6b00-8c61-4e36-b7e5-9098d270452f" (UID: "c48d6b00-8c61-4e36-b7e5-9098d270452f"). InnerVolumeSpecName "kube-api-access-rw7gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:22:11.893806 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.893742 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48d6b00-8c61-4e36-b7e5-9098d270452f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c48d6b00-8c61-4e36-b7e5-9098d270452f" (UID: "c48d6b00-8c61-4e36-b7e5-9098d270452f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:22:11.992412 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.992377 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48d6b00-8c61-4e36-b7e5-9098d270452f-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:22:11.992412 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.992404 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw7gt\" (UniqueName: \"kubernetes.io/projected/c48d6b00-8c61-4e36-b7e5-9098d270452f-kube-api-access-rw7gt\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:22:11.992412 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:11.992418 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48d6b00-8c61-4e36-b7e5-9098d270452f-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:22:12.717036 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:12.717008 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp" Jul 09 01:22:12.737231 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:12.737209 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp"] Jul 09 01:22:12.741336 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:12.741313 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b5edbb9-7bcdfbbdc-8bvzp"] Jul 09 01:22:12.927973 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:12.927939 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c48d6b00-8c61-4e36-b7e5-9098d270452f" path="/var/lib/kubelet/pods/c48d6b00-8c61-4e36-b7e5-9098d270452f/volumes" Jul 09 01:22:17.153863 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:17.153824 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:17.253192 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:17.253162 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:17.429806 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:17.429727 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:17.789475 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:17.789396 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:17.885322 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:17.885285 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:17.986154 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:17.986124 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:23.683403 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:23.683365 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:23.787735 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:23.787699 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:23.886190 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:23.886152 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:24.020359 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:24.020329 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:29.532543 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:29.532441 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:29.593413 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:29.593375 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:36.292828 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:36.292796 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:36.611043 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:36.610953 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:37.221886 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:37.221851 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:39.481189 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.481145 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:39.558657 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.558616 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz"] Jul 09 01:22:39.558957 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.558941 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48d6b00-8c61-4e36-b7e5-9098d270452f" containerName="maas-api" Jul 09 01:22:39.558957 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.558956 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48d6b00-8c61-4e36-b7e5-9098d270452f" containerName="maas-api" Jul 09 01:22:39.559103 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.559018 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48d6b00-8c61-4e36-b7e5-9098d270452f" containerName="maas-api" Jul 09 01:22:39.561877 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.561859 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.564254 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.564231 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:22:39.564355 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.564338 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:22:39.565033 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.565017 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e0f06239-serving-cert\"" Jul 09 01:22:39.565112 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.565044 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:22:39.570259 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.570214 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:39.571271 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.571252 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz"] Jul 09 01:22:39.626395 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.626364 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxgl\" (UniqueName: \"kubernetes.io/projected/4981376b-df7f-46c1-bb3d-b914353b49c3-kube-api-access-jmxgl\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.626546 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.626417 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981376b-df7f-46c1-bb3d-b914353b49c3-openshift-service-ca\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.626546 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.626447 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.634276 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.634252 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:39.727114 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.727084 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxgl\" (UniqueName: \"kubernetes.io/projected/4981376b-df7f-46c1-bb3d-b914353b49c3-kube-api-access-jmxgl\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.727272 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.727123 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981376b-df7f-46c1-bb3d-b914353b49c3-openshift-service-ca\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.727366 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.727339 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.727495 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:22:39.727474 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-serving-cert: secret "maas-api-e2e-mt-e0f06239-serving-cert" not found Jul 09 01:22:39.727566 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:22:39.727555 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls podName:4981376b-df7f-46c1-bb3d-b914353b49c3 nodeName:}" failed. No retries permitted until 2026-07-09 01:22:40.227537277 +0000 UTC m=+1624.505375907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls") pod "maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" (UID: "4981376b-df7f-46c1-bb3d-b914353b49c3") : secret "maas-api-e2e-mt-e0f06239-serving-cert" not found Jul 09 01:22:39.727863 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.727842 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981376b-df7f-46c1-bb3d-b914353b49c3-openshift-service-ca\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.736288 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.736234 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxgl\" (UniqueName: \"kubernetes.io/projected/4981376b-df7f-46c1-bb3d-b914353b49c3-kube-api-access-jmxgl\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:39.913094 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:39.913062 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:40.124701 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:40.124588 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:40.224536 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:40.224504 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:40.230349 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:40.230328 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:40.232617 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:40.232596 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls\") pod \"maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:40.472736 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:40.472695 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:40.601375 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:40.601350 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz"] Jul 09 01:22:40.603517 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:22:40.603480 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4981376b_df7f_46c1_bb3d_b914353b49c3.slice/crio-3d71faedf45acd413232b3ed04224772d8915efe886b5dbfa24538beb7a468c5 WatchSource:0}: Error finding container 3d71faedf45acd413232b3ed04224772d8915efe886b5dbfa24538beb7a468c5: Status 404 returned error can't find the container with id 3d71faedf45acd413232b3ed04224772d8915efe886b5dbfa24538beb7a468c5 Jul 09 01:22:40.833002 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:40.832929 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" event={"ID":"4981376b-df7f-46c1-bb3d-b914353b49c3","Type":"ContainerStarted","Data":"3d71faedf45acd413232b3ed04224772d8915efe886b5dbfa24538beb7a468c5"} Jul 09 01:22:41.838693 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:41.838638 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" event={"ID":"4981376b-df7f-46c1-bb3d-b914353b49c3","Type":"ContainerStarted","Data":"6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199"} Jul 09 01:22:41.839201 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:41.838778 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:22:41.856202 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:41.856154 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" podStartSLOduration=2.5543201570000003 podStartE2EDuration="2.856139894s" totalImagesPullingTime="301.819737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:22:41.854852203 +0000 UTC m=+1626.132690855" watchObservedRunningTime="2026-07-09 01:22:41.856139894 +0000 UTC m=+1626.133978544" Jul 09 01:22:43.969738 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:43.969702 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:47.386104 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:47.386068 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:47.486622 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:47.486568 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:22:47.848883 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:22:47.848856 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:23:01.685998 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:01.685956 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:23:01.789254 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:01.789215 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:23:14.658533 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:14.658489 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:23:36.761947 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:36.761907 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:23:37.072997 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.072925 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz"] Jul 09 01:23:37.073173 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.073143 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" podUID="4981376b-df7f-46c1-bb3d-b914353b49c3" containerName="maas-api" containerID="cri-o://6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199" gracePeriod=30 Jul 09 01:23:37.106549 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.106520 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:23:37.320304 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.320277 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:23:37.385276 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.385199 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls\") pod \"4981376b-df7f-46c1-bb3d-b914353b49c3\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " Jul 09 01:23:37.385276 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.385241 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981376b-df7f-46c1-bb3d-b914353b49c3-openshift-service-ca\") pod \"4981376b-df7f-46c1-bb3d-b914353b49c3\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " Jul 09 01:23:37.385455 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.385279 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmxgl\" (UniqueName: \"kubernetes.io/projected/4981376b-df7f-46c1-bb3d-b914353b49c3-kube-api-access-jmxgl\") pod \"4981376b-df7f-46c1-bb3d-b914353b49c3\" (UID: \"4981376b-df7f-46c1-bb3d-b914353b49c3\") " Jul 09 01:23:37.385659 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.385596 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4981376b-df7f-46c1-bb3d-b914353b49c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4981376b-df7f-46c1-bb3d-b914353b49c3" (UID: "4981376b-df7f-46c1-bb3d-b914353b49c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:23:37.387336 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.387305 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4981376b-df7f-46c1-bb3d-b914353b49c3" (UID: "4981376b-df7f-46c1-bb3d-b914353b49c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:23:37.387418 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.387348 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4981376b-df7f-46c1-bb3d-b914353b49c3-kube-api-access-jmxgl" (OuterVolumeSpecName: "kube-api-access-jmxgl") pod "4981376b-df7f-46c1-bb3d-b914353b49c3" (UID: "4981376b-df7f-46c1-bb3d-b914353b49c3"). InnerVolumeSpecName "kube-api-access-jmxgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:23:37.486108 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.486072 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981376b-df7f-46c1-bb3d-b914353b49c3-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:23:37.486108 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.486100 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981376b-df7f-46c1-bb3d-b914353b49c3-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:23:37.486108 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:37.486110 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxgl\" (UniqueName: \"kubernetes.io/projected/4981376b-df7f-46c1-bb3d-b914353b49c3-kube-api-access-jmxgl\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:23:38.065701 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.065636 2543 generic.go:356] "Generic (PLEG): container finished" podID="4981376b-df7f-46c1-bb3d-b914353b49c3" containerID="6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199" exitCode=0 Jul 09 01:23:38.066142 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.065704 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" Jul 09 01:23:38.066142 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.065724 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" event={"ID":"4981376b-df7f-46c1-bb3d-b914353b49c3","Type":"ContainerDied","Data":"6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199"} Jul 09 01:23:38.066142 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.065776 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz" event={"ID":"4981376b-df7f-46c1-bb3d-b914353b49c3","Type":"ContainerDied","Data":"3d71faedf45acd413232b3ed04224772d8915efe886b5dbfa24538beb7a468c5"} Jul 09 01:23:38.066142 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.065807 2543 scope.go:117] "RemoveContainer" containerID="6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199" Jul 09 01:23:38.075019 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.074872 2543 scope.go:117] "RemoveContainer" containerID="6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199" Jul 09 01:23:38.075098 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:23:38.075080 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199\": container with ID starting with 6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199 not found: ID does not exist" containerID="6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199" Jul 09 01:23:38.075144 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.075106 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199"} err="failed to get container status \"6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199\": rpc error: code = NotFound desc = could not find container \"6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199\": container with ID starting with 6470f4728119bfe2157f25627bacabb45250b04ac378cbfaa5154726dbc4b199 not found: ID does not exist" Jul 09 01:23:38.085907 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.085887 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz"] Jul 09 01:23:38.088253 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.088230 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0f06239-64d5cdf7b9-jg9vz"] Jul 09 01:23:38.928552 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:38.928517 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4981376b-df7f-46c1-bb3d-b914353b49c3" path="/var/lib/kubelet/pods/4981376b-df7f-46c1-bb3d-b914353b49c3/volumes" Jul 09 01:23:42.839488 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:42.839455 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:23:47.919716 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:23:47.919684 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:24.564757 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:24.564724 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:24.878983 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:24.878896 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:30.316911 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:30.316870 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:30.417355 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:30.417316 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:30.602156 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:30.602074 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:31.876018 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:31.875977 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:31.973365 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:31.973313 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:32.073199 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:32.073171 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:32.805393 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:32.805359 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:32.903786 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:32.903736 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:33.008307 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:33.008261 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:34.353535 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:34.353501 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:40.235550 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:40.235511 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:40.289702 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:40.289662 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:40.898015 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:40.897976 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:46.545583 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:46.545539 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:46.610730 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:46.610690 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:46.687432 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:46.687391 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:46.782816 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:46.782774 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:57.255062 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:57.255027 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:24:57.618578 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:24:57.618482 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:18.570554 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:18.570516 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:24.231793 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:24.231752 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:24.295520 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:24.295491 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:24.577876 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:24.577771 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:24.865299 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:24.865219 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:24.967508 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:24.967473 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:28.286138 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.286052 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:28.356064 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.356034 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb"] Jul 09 01:25:28.356347 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.356336 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4981376b-df7f-46c1-bb3d-b914353b49c3" containerName="maas-api" Jul 09 01:25:28.356398 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.356351 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="4981376b-df7f-46c1-bb3d-b914353b49c3" containerName="maas-api" Jul 09 01:25:28.356443 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.356406 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="4981376b-df7f-46c1-bb3d-b914353b49c3" containerName="maas-api" Jul 09 01:25:28.359186 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.359169 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.361522 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.361485 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3460b5-serving-cert\"" Jul 09 01:25:28.361522 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.361508 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:25:28.361729 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.361508 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:25:28.362297 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.362277 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:25:28.369625 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.369603 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb"] Jul 09 01:25:28.375497 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.375473 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:28.422891 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.422873 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4rxc\" (UniqueName: \"kubernetes.io/projected/01a54812-6151-4191-87d0-5386bb913232-kube-api-access-x4rxc\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.422986 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.422902 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a54812-6151-4191-87d0-5386bb913232-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.422986 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.422936 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.432982 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.432942 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:28.524006 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.523983 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4rxc\" (UniqueName: \"kubernetes.io/projected/01a54812-6151-4191-87d0-5386bb913232-kube-api-access-x4rxc\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.524127 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.524015 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a54812-6151-4191-87d0-5386bb913232-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.524127 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.524056 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.524200 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:25:28.524173 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-serving-cert: secret "maas-api-e2e-inf-a-3460b5-serving-cert" not found Jul 09 01:25:28.524246 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:25:28.524235 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls podName:01a54812-6151-4191-87d0-5386bb913232 nodeName:}" failed. No retries permitted until 2026-07-09 01:25:29.024219336 +0000 UTC m=+1793.302057972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls") pod "maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" (UID: "01a54812-6151-4191-87d0-5386bb913232") : secret "maas-api-e2e-inf-a-3460b5-serving-cert" not found Jul 09 01:25:28.524612 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.524594 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a54812-6151-4191-87d0-5386bb913232-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.531972 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.531948 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4rxc\" (UniqueName: \"kubernetes.io/projected/01a54812-6151-4191-87d0-5386bb913232-kube-api-access-x4rxc\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:28.807336 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.807296 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:28.907047 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:28.907020 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:29.006712 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:29.006679 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:29.027791 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:29.027762 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:29.030046 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:29.030023 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls\") pod \"maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:29.269916 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:29.269892 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:29.395572 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:29.391297 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb"] Jul 09 01:25:29.397245 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:29.397223 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 01:25:29.519064 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:29.519035 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" event={"ID":"01a54812-6151-4191-87d0-5386bb913232","Type":"ContainerStarted","Data":"d1de72a427d68c4bc41ecf7b0b58d3556105f1a6662a438625fa8ba3e2f65be1"} Jul 09 01:25:30.525390 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:30.525356 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" event={"ID":"01a54812-6151-4191-87d0-5386bb913232","Type":"ContainerStarted","Data":"0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9"} Jul 09 01:25:30.525793 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:30.525448 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:30.541940 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:30.541899 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" podStartSLOduration=2.105756218 podStartE2EDuration="2.541884899s" totalImagesPullingTime="436.128681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:25:30.540473948 +0000 UTC m=+1794.818312590" watchObservedRunningTime="2026-07-09 01:25:30.541884899 +0000 UTC m=+1794.819723549" Jul 09 01:25:32.789985 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:32.789946 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:33.115905 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:33.115825 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:33.435351 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:33.435267 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:33.537850 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:33.537805 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:36.262749 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.262712 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:36.364248 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.364216 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:36.493295 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.493266 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm"] Jul 09 01:25:36.497514 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.497491 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.499893 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.499872 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-adaf0e-serving-cert\"" Jul 09 01:25:36.506904 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.506852 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm"] Jul 09 01:25:36.524961 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.524906 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:36.535379 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.535359 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:25:36.589679 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.589628 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65537203-a8d0-4e5b-9383-186cca6097b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.589806 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.589754 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65537203-a8d0-4e5b-9383-186cca6097b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.589806 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.589787 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22lq\" (UniqueName: \"kubernetes.io/projected/65537203-a8d0-4e5b-9383-186cca6097b2-kube-api-access-k22lq\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.637454 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.637421 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:36.690549 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.690521 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65537203-a8d0-4e5b-9383-186cca6097b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.690709 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.690597 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65537203-a8d0-4e5b-9383-186cca6097b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.690709 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.690628 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22lq\" (UniqueName: \"kubernetes.io/projected/65537203-a8d0-4e5b-9383-186cca6097b2-kube-api-access-k22lq\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.691329 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.691307 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65537203-a8d0-4e5b-9383-186cca6097b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.692978 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.692956 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65537203-a8d0-4e5b-9383-186cca6097b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.698908 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.698889 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22lq\" (UniqueName: \"kubernetes.io/projected/65537203-a8d0-4e5b-9383-186cca6097b2-kube-api-access-k22lq\") pod \"maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.811448 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.811383 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:36.936711 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:36.936683 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm"] Jul 09 01:25:36.939002 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:25:36.938977 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65537203_a8d0_4e5b_9383_186cca6097b2.slice/crio-ac937cd85b301978a8b78dafa72a18bd11069d86dc2a9cd743f6b95f350516e6 WatchSource:0}: Error finding container ac937cd85b301978a8b78dafa72a18bd11069d86dc2a9cd743f6b95f350516e6: Status 404 returned error can't find the container with id ac937cd85b301978a8b78dafa72a18bd11069d86dc2a9cd743f6b95f350516e6 Jul 09 01:25:37.120067 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:37.119991 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:37.227285 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:37.227259 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:37.557906 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:37.557877 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" event={"ID":"65537203-a8d0-4e5b-9383-186cca6097b2","Type":"ContainerStarted","Data":"f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030"} Jul 09 01:25:37.558240 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:37.557925 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" event={"ID":"65537203-a8d0-4e5b-9383-186cca6097b2","Type":"ContainerStarted","Data":"ac937cd85b301978a8b78dafa72a18bd11069d86dc2a9cd743f6b95f350516e6"} Jul 09 01:25:37.558240 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:37.557997 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:37.573079 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:37.573015 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" podStartSLOduration=1.271360676 podStartE2EDuration="1.573001942s" totalImagesPullingTime="301.641266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:25:37.571818367 +0000 UTC m=+1801.849657031" watchObservedRunningTime="2026-07-09 01:25:37.573001942 +0000 UTC m=+1801.850840591" Jul 09 01:25:40.764089 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:40.764053 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:43.567810 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:43.567782 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:25:50.497621 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:50.497592 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:50.598679 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:50.598628 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:55.891091 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:55.891054 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:57.826214 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:57.826180 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:25:57.927032 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:25:57.926999 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:19.549520 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:19.549480 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:41.954447 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:41.954387 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:44.631315 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:44.631285 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:45.572123 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:45.572088 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:45.719200 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:45.719167 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:46.117780 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.117747 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:46.222808 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.222778 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:46.320370 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.320333 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:46.413941 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.413859 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb"] Jul 09 01:26:46.414146 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.414120 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" podUID="01a54812-6151-4191-87d0-5386bb913232" containerName="maas-api" containerID="cri-o://0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9" gracePeriod=30 Jul 09 01:26:46.435038 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.435015 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:26:46.530329 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.530289 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" podUID="01a54812-6151-4191-87d0-5386bb913232" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.29:8443/health\": dial tcp 10.134.0.29:8443: connect: connection refused" Jul 09 01:26:46.657227 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.657200 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:26:46.732409 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.732381 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4rxc\" (UniqueName: \"kubernetes.io/projected/01a54812-6151-4191-87d0-5386bb913232-kube-api-access-x4rxc\") pod \"01a54812-6151-4191-87d0-5386bb913232\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " Jul 09 01:26:46.732841 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.732465 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a54812-6151-4191-87d0-5386bb913232-openshift-service-ca\") pod \"01a54812-6151-4191-87d0-5386bb913232\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " Jul 09 01:26:46.732841 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.732498 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls\") pod \"01a54812-6151-4191-87d0-5386bb913232\" (UID: \"01a54812-6151-4191-87d0-5386bb913232\") " Jul 09 01:26:46.732917 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.732830 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a54812-6151-4191-87d0-5386bb913232-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01a54812-6151-4191-87d0-5386bb913232" (UID: "01a54812-6151-4191-87d0-5386bb913232"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:26:46.734448 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.734422 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01a54812-6151-4191-87d0-5386bb913232" (UID: "01a54812-6151-4191-87d0-5386bb913232"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:26:46.734535 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.734444 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a54812-6151-4191-87d0-5386bb913232-kube-api-access-x4rxc" (OuterVolumeSpecName: "kube-api-access-x4rxc") pod "01a54812-6151-4191-87d0-5386bb913232" (UID: "01a54812-6151-4191-87d0-5386bb913232"). InnerVolumeSpecName "kube-api-access-x4rxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:26:46.833776 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.833749 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4rxc\" (UniqueName: \"kubernetes.io/projected/01a54812-6151-4191-87d0-5386bb913232-kube-api-access-x4rxc\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:26:46.833776 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.833772 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a54812-6151-4191-87d0-5386bb913232-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:26:46.833922 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.833782 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a54812-6151-4191-87d0-5386bb913232-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:26:46.855566 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.855541 2543 generic.go:356] "Generic (PLEG): container finished" podID="01a54812-6151-4191-87d0-5386bb913232" containerID="0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9" exitCode=0 Jul 09 01:26:46.855693 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.855585 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" Jul 09 01:26:46.855693 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.855626 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" event={"ID":"01a54812-6151-4191-87d0-5386bb913232","Type":"ContainerDied","Data":"0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9"} Jul 09 01:26:46.855773 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.855694 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb" event={"ID":"01a54812-6151-4191-87d0-5386bb913232","Type":"ContainerDied","Data":"d1de72a427d68c4bc41ecf7b0b58d3556105f1a6662a438625fa8ba3e2f65be1"} Jul 09 01:26:46.855773 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.855717 2543 scope.go:117] "RemoveContainer" containerID="0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9" Jul 09 01:26:46.863907 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.863892 2543 scope.go:117] "RemoveContainer" containerID="0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9" Jul 09 01:26:46.864162 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:26:46.864144 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9\": container with ID starting with 0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9 not found: ID does not exist" containerID="0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9" Jul 09 01:26:46.864215 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.864169 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9"} err="failed to get container status \"0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9\": rpc error: code = NotFound desc = could not find container \"0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9\": container with ID starting with 0ee7ed062268ca52ad77d03bdb5f12da3d59d4431937e644cbba8f7595dad3c9 not found: ID does not exist" Jul 09 01:26:46.875428 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.875407 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb"] Jul 09 01:26:46.881255 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.881235 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3460b5-6b5945bc64-9hnmb"] Jul 09 01:26:46.927805 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:46.927784 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a54812-6151-4191-87d0-5386bb913232" path="/var/lib/kubelet/pods/01a54812-6151-4191-87d0-5386bb913232/volumes" Jul 09 01:26:51.199852 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:26:51.199822 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:27.835244 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:27.835197 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:28.930066 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:28.930032 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:29.031013 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.030980 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:29.130969 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.130938 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:29.230975 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.230952 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:29.292870 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.292842 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm"] Jul 09 01:27:29.293108 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.293083 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" podUID="65537203-a8d0-4e5b-9383-186cca6097b2" containerName="maas-api" containerID="cri-o://f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030" gracePeriod=30 Jul 09 01:27:29.330759 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.330729 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:29.743201 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.743173 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:27:29.844069 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.843990 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k22lq\" (UniqueName: \"kubernetes.io/projected/65537203-a8d0-4e5b-9383-186cca6097b2-kube-api-access-k22lq\") pod \"65537203-a8d0-4e5b-9383-186cca6097b2\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " Jul 09 01:27:29.844235 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.844078 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65537203-a8d0-4e5b-9383-186cca6097b2-maas-api-tls\") pod \"65537203-a8d0-4e5b-9383-186cca6097b2\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " Jul 09 01:27:29.844235 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.844110 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65537203-a8d0-4e5b-9383-186cca6097b2-openshift-service-ca\") pod \"65537203-a8d0-4e5b-9383-186cca6097b2\" (UID: \"65537203-a8d0-4e5b-9383-186cca6097b2\") " Jul 09 01:27:29.844503 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.844478 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65537203-a8d0-4e5b-9383-186cca6097b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65537203-a8d0-4e5b-9383-186cca6097b2" (UID: "65537203-a8d0-4e5b-9383-186cca6097b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:27:29.846054 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.846024 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65537203-a8d0-4e5b-9383-186cca6097b2-kube-api-access-k22lq" (OuterVolumeSpecName: "kube-api-access-k22lq") pod "65537203-a8d0-4e5b-9383-186cca6097b2" (UID: "65537203-a8d0-4e5b-9383-186cca6097b2"). InnerVolumeSpecName "kube-api-access-k22lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:27:29.846149 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.846092 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65537203-a8d0-4e5b-9383-186cca6097b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65537203-a8d0-4e5b-9383-186cca6097b2" (UID: "65537203-a8d0-4e5b-9383-186cca6097b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:27:29.944530 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.944502 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22lq\" (UniqueName: \"kubernetes.io/projected/65537203-a8d0-4e5b-9383-186cca6097b2-kube-api-access-k22lq\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:27:29.944530 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.944528 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65537203-a8d0-4e5b-9383-186cca6097b2-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:27:29.944908 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:29.944557 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65537203-a8d0-4e5b-9383-186cca6097b2-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:27:30.031054 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.031025 2543 generic.go:356] "Generic (PLEG): container finished" podID="65537203-a8d0-4e5b-9383-186cca6097b2" containerID="f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030" exitCode=0 Jul 09 01:27:30.031177 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.031067 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" Jul 09 01:27:30.031177 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.031111 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" event={"ID":"65537203-a8d0-4e5b-9383-186cca6097b2","Type":"ContainerDied","Data":"f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030"} Jul 09 01:27:30.031177 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.031159 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm" event={"ID":"65537203-a8d0-4e5b-9383-186cca6097b2","Type":"ContainerDied","Data":"ac937cd85b301978a8b78dafa72a18bd11069d86dc2a9cd743f6b95f350516e6"} Jul 09 01:27:30.031289 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.031182 2543 scope.go:117] "RemoveContainer" containerID="f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030" Jul 09 01:27:30.040064 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.040045 2543 scope.go:117] "RemoveContainer" containerID="f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030" Jul 09 01:27:30.042956 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:27:30.042888 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030\": container with ID starting with f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030 not found: ID does not exist" containerID="f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030" Jul 09 01:27:30.042956 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.042921 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030"} err="failed to get container status \"f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030\": rpc error: code = NotFound desc = could not find container \"f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030\": container with ID starting with f8d39353e18cdfbc3c8a48bd83f29ac6df46e54bb9fac30504e735fe58abc030 not found: ID does not exist" Jul 09 01:27:30.052009 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.051986 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm"] Jul 09 01:27:30.056426 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.056403 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adaf0e-5c9c749d6-7fjhm"] Jul 09 01:27:30.929153 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:30.929111 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65537203-a8d0-4e5b-9383-186cca6097b2" path="/var/lib/kubelet/pods/65537203-a8d0-4e5b-9383-186cca6097b2/volumes" Jul 09 01:27:34.730427 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:34.730392 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:34.834249 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:34.834211 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:35.001393 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:35.001302 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:35.570983 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:35.570949 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:35.897252 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:35.897166 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:36.030029 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:36.029991 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:38.893970 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.893938 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:38.967109 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.967079 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w"] Jul 09 01:27:38.967373 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.967361 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65537203-a8d0-4e5b-9383-186cca6097b2" containerName="maas-api" Jul 09 01:27:38.967416 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.967374 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="65537203-a8d0-4e5b-9383-186cca6097b2" containerName="maas-api" Jul 09 01:27:38.967416 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.967390 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a54812-6151-4191-87d0-5386bb913232" containerName="maas-api" Jul 09 01:27:38.967416 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.967396 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a54812-6151-4191-87d0-5386bb913232" containerName="maas-api" Jul 09 01:27:38.967509 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.967444 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="65537203-a8d0-4e5b-9383-186cca6097b2" containerName="maas-api" Jul 09 01:27:38.967509 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.967453 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a54812-6151-4191-87d0-5386bb913232" containerName="maas-api" Jul 09 01:27:38.971894 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.971871 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:38.974467 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.974446 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:27:38.974587 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.974446 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-45d60b-serving-cert\"" Jul 09 01:27:38.975356 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.975338 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:27:38.975455 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.975339 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:27:38.980209 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.980190 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w"] Jul 09 01:27:38.982944 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:38.982918 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:39.019118 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.019096 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjk4b\" (UniqueName: \"kubernetes.io/projected/04e560fe-d980-4746-babd-d69527fded5c-kube-api-access-jjk4b\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.019214 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.019167 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.019269 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.019239 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e560fe-d980-4746-babd-d69527fded5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.039478 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.039457 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:39.120501 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.120473 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.120624 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.120531 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e560fe-d980-4746-babd-d69527fded5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.120624 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.120571 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjk4b\" (UniqueName: \"kubernetes.io/projected/04e560fe-d980-4746-babd-d69527fded5c-kube-api-access-jjk4b\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.120624 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:27:39.120614 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-serving-cert: secret "maas-api-e2e-api-a-45d60b-serving-cert" not found Jul 09 01:27:39.120838 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:27:39.120698 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls podName:04e560fe-d980-4746-babd-d69527fded5c nodeName:}" failed. No retries permitted until 2026-07-09 01:27:39.620676577 +0000 UTC m=+1923.898515219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls") pod "maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" (UID: "04e560fe-d980-4746-babd-d69527fded5c") : secret "maas-api-e2e-api-a-45d60b-serving-cert" not found Jul 09 01:27:39.121219 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.121202 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e560fe-d980-4746-babd-d69527fded5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.128819 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.128800 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjk4b\" (UniqueName: \"kubernetes.io/projected/04e560fe-d980-4746-babd-d69527fded5c-kube-api-access-jjk4b\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.310841 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.310811 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:39.443632 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.443599 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:39.543124 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.543094 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:39.625338 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.625261 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.627577 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.627550 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls\") pod \"maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:39.884202 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:39.884115 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:40.023884 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:40.023860 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w"] Jul 09 01:27:40.026285 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:27:40.026257 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e560fe_d980_4746_babd_d69527fded5c.slice/crio-ed71b185de538810f0163fbe48c1709aa84d3afc9e1861749af00364b27dba6d WatchSource:0}: Error finding container ed71b185de538810f0163fbe48c1709aa84d3afc9e1861749af00364b27dba6d: Status 404 returned error can't find the container with id ed71b185de538810f0163fbe48c1709aa84d3afc9e1861749af00364b27dba6d Jul 09 01:27:40.072707 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:40.072673 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" event={"ID":"04e560fe-d980-4746-babd-d69527fded5c","Type":"ContainerStarted","Data":"ed71b185de538810f0163fbe48c1709aa84d3afc9e1861749af00364b27dba6d"} Jul 09 01:27:41.077855 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:41.077820 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" event={"ID":"04e560fe-d980-4746-babd-d69527fded5c","Type":"ContainerStarted","Data":"a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36"} Jul 09 01:27:41.078234 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:41.077894 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:41.094181 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:41.094132 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" podStartSLOduration=2.735661169 podStartE2EDuration="3.094115508s" totalImagesPullingTime="358.454339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:27:41.092984985 +0000 UTC m=+1925.370823636" watchObservedRunningTime="2026-07-09 01:27:41.094115508 +0000 UTC m=+1925.371954159" Jul 09 01:27:42.247423 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:42.247374 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:43.390972 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:43.390926 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:43.879149 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:43.879088 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:44.205277 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:44.205243 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:44.963996 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:44.963959 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:46.919388 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:46.919345 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:47.020319 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.020290 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:47.087619 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.087592 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:27:47.109668 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.107974 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn"] Jul 09 01:27:47.116596 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.116572 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.120999 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.120966 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fbc435-serving-cert\"" Jul 09 01:27:47.122427 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.122403 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn"] Jul 09 01:27:47.158109 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.158081 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:47.188582 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.188531 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.188698 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.188592 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d981410-5bdf-4f71-a44a-7c47ab837672-openshift-service-ca\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.188768 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.188749 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2t47\" (UniqueName: \"kubernetes.io/projected/3d981410-5bdf-4f71-a44a-7c47ab837672-kube-api-access-x2t47\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.289142 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.289114 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2t47\" (UniqueName: \"kubernetes.io/projected/3d981410-5bdf-4f71-a44a-7c47ab837672-kube-api-access-x2t47\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.289297 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.289158 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.289297 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.289219 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d981410-5bdf-4f71-a44a-7c47ab837672-openshift-service-ca\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.289411 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:27:47.289338 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-serving-cert: secret "maas-api-e2e-api-b-fbc435-serving-cert" not found Jul 09 01:27:47.289462 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:27:47.289419 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls podName:3d981410-5bdf-4f71-a44a-7c47ab837672 nodeName:}" failed. No retries permitted until 2026-07-09 01:27:47.789398029 +0000 UTC m=+1932.067236669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls") pod "maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" (UID: "3d981410-5bdf-4f71-a44a-7c47ab837672") : secret "maas-api-e2e-api-b-fbc435-serving-cert" not found Jul 09 01:27:47.289897 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.289878 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d981410-5bdf-4f71-a44a-7c47ab837672-openshift-service-ca\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.297711 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.297693 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2t47\" (UniqueName: \"kubernetes.io/projected/3d981410-5bdf-4f71-a44a-7c47ab837672-kube-api-access-x2t47\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.409461 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.409429 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:47.689268 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.689230 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:47.771777 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.771745 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:47.792731 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.792710 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:47.795197 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:47.795172 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls\") pod \"maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:48.029596 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:48.029558 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:48.159529 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:48.159504 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn"] Jul 09 01:27:48.162272 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:27:48.162248 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d981410_5bdf_4f71_a44a_7c47ab837672.slice/crio-e8740aaceb43b6f1340cdc6c5688620fe1ce3542d0c678934320bb0dc12c1170 WatchSource:0}: Error finding container e8740aaceb43b6f1340cdc6c5688620fe1ce3542d0c678934320bb0dc12c1170: Status 404 returned error can't find the container with id e8740aaceb43b6f1340cdc6c5688620fe1ce3542d0c678934320bb0dc12c1170 Jul 09 01:27:48.487708 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:48.487668 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:48.587961 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:48.587932 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:49.110279 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:49.110243 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" event={"ID":"3d981410-5bdf-4f71-a44a-7c47ab837672","Type":"ContainerStarted","Data":"dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e"} Jul 09 01:27:49.110636 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:49.110300 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" event={"ID":"3d981410-5bdf-4f71-a44a-7c47ab837672","Type":"ContainerStarted","Data":"e8740aaceb43b6f1340cdc6c5688620fe1ce3542d0c678934320bb0dc12c1170"} Jul 09 01:27:49.110636 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:49.110380 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:49.125907 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:49.125866 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" podStartSLOduration=1.734641366 podStartE2EDuration="2.125849775s" totalImagesPullingTime="391.208409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:27:49.124600102 +0000 UTC m=+1933.402438754" watchObservedRunningTime="2026-07-09 01:27:49.125849775 +0000 UTC m=+1933.403688430" Jul 09 01:27:49.177222 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:49.177193 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:49.278757 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:49.278730 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:51.452240 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:51.452207 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:27:55.119165 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:55.119133 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:27:59.696559 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:27:59.696524 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:00.137453 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.137373 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:00.234734 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.234708 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:00.335568 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.335540 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:00.430891 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.430805 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w"] Jul 09 01:28:00.431045 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.431019 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" podUID="04e560fe-d980-4746-babd-d69527fded5c" containerName="maas-api" containerID="cri-o://a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36" gracePeriod=30 Jul 09 01:28:00.457260 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.457231 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:00.683240 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.683182 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:28:00.800444 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.800412 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls\") pod \"04e560fe-d980-4746-babd-d69527fded5c\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " Jul 09 01:28:00.800914 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.800469 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjk4b\" (UniqueName: \"kubernetes.io/projected/04e560fe-d980-4746-babd-d69527fded5c-kube-api-access-jjk4b\") pod \"04e560fe-d980-4746-babd-d69527fded5c\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " Jul 09 01:28:00.800914 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.800501 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e560fe-d980-4746-babd-d69527fded5c-openshift-service-ca\") pod \"04e560fe-d980-4746-babd-d69527fded5c\" (UID: \"04e560fe-d980-4746-babd-d69527fded5c\") " Jul 09 01:28:00.801001 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.800946 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e560fe-d980-4746-babd-d69527fded5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04e560fe-d980-4746-babd-d69527fded5c" (UID: "04e560fe-d980-4746-babd-d69527fded5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:28:00.802482 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.802450 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04e560fe-d980-4746-babd-d69527fded5c" (UID: "04e560fe-d980-4746-babd-d69527fded5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:28:00.802614 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.802591 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e560fe-d980-4746-babd-d69527fded5c-kube-api-access-jjk4b" (OuterVolumeSpecName: "kube-api-access-jjk4b") pod "04e560fe-d980-4746-babd-d69527fded5c" (UID: "04e560fe-d980-4746-babd-d69527fded5c"). InnerVolumeSpecName "kube-api-access-jjk4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:28:00.901214 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.901186 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e560fe-d980-4746-babd-d69527fded5c-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:28:00.901214 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.901210 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjk4b\" (UniqueName: \"kubernetes.io/projected/04e560fe-d980-4746-babd-d69527fded5c-kube-api-access-jjk4b\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:28:00.901381 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:00.901221 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e560fe-d980-4746-babd-d69527fded5c-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:28:01.160861 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.160826 2543 generic.go:356] "Generic (PLEG): container finished" podID="04e560fe-d980-4746-babd-d69527fded5c" containerID="a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36" exitCode=0 Jul 09 01:28:01.161038 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.160870 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" Jul 09 01:28:01.161038 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.160899 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" event={"ID":"04e560fe-d980-4746-babd-d69527fded5c","Type":"ContainerDied","Data":"a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36"} Jul 09 01:28:01.161038 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.160958 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w" event={"ID":"04e560fe-d980-4746-babd-d69527fded5c","Type":"ContainerDied","Data":"ed71b185de538810f0163fbe48c1709aa84d3afc9e1861749af00364b27dba6d"} Jul 09 01:28:01.161038 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.160979 2543 scope.go:117] "RemoveContainer" containerID="a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36" Jul 09 01:28:01.169276 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.169258 2543 scope.go:117] "RemoveContainer" containerID="a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36" Jul 09 01:28:01.169518 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:01.169499 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36\": container with ID starting with a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36 not found: ID does not exist" containerID="a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36" Jul 09 01:28:01.169557 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.169527 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36"} err="failed to get container status \"a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36\": rpc error: code = NotFound desc = could not find container \"a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36\": container with ID starting with a867806e3e333073f41649db09549f88220700ebc28902cb380843b3a70f1a36 not found: ID does not exist" Jul 09 01:28:01.175517 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.175494 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w"] Jul 09 01:28:01.180753 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:01.180733 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45d60b-b8f66cdbb-q4g7w"] Jul 09 01:28:02.927749 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:02.927713 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e560fe-d980-4746-babd-d69527fded5c" path="/var/lib/kubelet/pods/04e560fe-d980-4746-babd-d69527fded5c/volumes" Jul 09 01:28:06.032385 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:06.032347 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:06.134109 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:06.134085 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:06.291173 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:06.291098 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:06.662852 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:06.662823 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:07.061244 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.061201 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:07.162721 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.162677 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:07.262329 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.262297 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:07.323295 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.323206 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn"] Jul 09 01:28:07.323478 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.323444 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" podUID="3d981410-5bdf-4f71-a44a-7c47ab837672" containerName="maas-api" containerID="cri-o://dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e" gracePeriod=30 Jul 09 01:28:07.361060 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.361023 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:07.558935 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.558913 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:28:07.655054 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.654979 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2t47\" (UniqueName: \"kubernetes.io/projected/3d981410-5bdf-4f71-a44a-7c47ab837672-kube-api-access-x2t47\") pod \"3d981410-5bdf-4f71-a44a-7c47ab837672\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " Jul 09 01:28:07.655054 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.655012 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls\") pod \"3d981410-5bdf-4f71-a44a-7c47ab837672\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " Jul 09 01:28:07.655054 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.655035 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d981410-5bdf-4f71-a44a-7c47ab837672-openshift-service-ca\") pod \"3d981410-5bdf-4f71-a44a-7c47ab837672\" (UID: \"3d981410-5bdf-4f71-a44a-7c47ab837672\") " Jul 09 01:28:07.655424 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.655402 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d981410-5bdf-4f71-a44a-7c47ab837672-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d981410-5bdf-4f71-a44a-7c47ab837672" (UID: "3d981410-5bdf-4f71-a44a-7c47ab837672"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:28:07.657106 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.657077 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d981410-5bdf-4f71-a44a-7c47ab837672" (UID: "3d981410-5bdf-4f71-a44a-7c47ab837672"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:28:07.657193 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.657142 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d981410-5bdf-4f71-a44a-7c47ab837672-kube-api-access-x2t47" (OuterVolumeSpecName: "kube-api-access-x2t47") pod "3d981410-5bdf-4f71-a44a-7c47ab837672" (UID: "3d981410-5bdf-4f71-a44a-7c47ab837672"). InnerVolumeSpecName "kube-api-access-x2t47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:28:07.756232 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.756204 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2t47\" (UniqueName: \"kubernetes.io/projected/3d981410-5bdf-4f71-a44a-7c47ab837672-kube-api-access-x2t47\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:28:07.756232 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.756228 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d981410-5bdf-4f71-a44a-7c47ab837672-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:28:07.756374 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:07.756238 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d981410-5bdf-4f71-a44a-7c47ab837672-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:28:08.190283 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.190248 2543 generic.go:356] "Generic (PLEG): container finished" podID="3d981410-5bdf-4f71-a44a-7c47ab837672" containerID="dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e" exitCode=0 Jul 09 01:28:08.190804 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.190299 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" Jul 09 01:28:08.190804 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.190334 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" event={"ID":"3d981410-5bdf-4f71-a44a-7c47ab837672","Type":"ContainerDied","Data":"dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e"} Jul 09 01:28:08.190804 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.190394 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn" event={"ID":"3d981410-5bdf-4f71-a44a-7c47ab837672","Type":"ContainerDied","Data":"e8740aaceb43b6f1340cdc6c5688620fe1ce3542d0c678934320bb0dc12c1170"} Jul 09 01:28:08.190804 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.190429 2543 scope.go:117] "RemoveContainer" containerID="dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e" Jul 09 01:28:08.199406 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.199391 2543 scope.go:117] "RemoveContainer" containerID="dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e" Jul 09 01:28:08.199692 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:08.199675 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e\": container with ID starting with dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e not found: ID does not exist" containerID="dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e" Jul 09 01:28:08.199748 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.199701 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e"} err="failed to get container status \"dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e\": rpc error: code = NotFound desc = could not find container \"dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e\": container with ID starting with dbed9f7355095dd28a77e409b413245f86aad213c4714d7222dde0c8bf7cf91e not found: ID does not exist" Jul 09 01:28:08.211021 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.210994 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn"] Jul 09 01:28:08.212998 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.212978 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fbc435-6f9cb54cc5-4sggn"] Jul 09 01:28:08.928475 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:08.928445 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d981410-5bdf-4f71-a44a-7c47ab837672" path="/var/lib/kubelet/pods/3d981410-5bdf-4f71-a44a-7c47ab837672/volumes" Jul 09 01:28:12.815883 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:12.815849 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:12.914902 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:12.914873 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:13.108764 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.108680 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:13.629287 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.629254 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:13.902885 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.902794 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h"] Jul 09 01:28:13.903300 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.903250 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e560fe-d980-4746-babd-d69527fded5c" containerName="maas-api" Jul 09 01:28:13.903300 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.903270 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e560fe-d980-4746-babd-d69527fded5c" containerName="maas-api" Jul 09 01:28:13.903300 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.903301 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d981410-5bdf-4f71-a44a-7c47ab837672" containerName="maas-api" Jul 09 01:28:13.903463 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.903309 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d981410-5bdf-4f71-a44a-7c47ab837672" containerName="maas-api" Jul 09 01:28:13.903463 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.903377 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e560fe-d980-4746-babd-d69527fded5c" containerName="maas-api" Jul 09 01:28:13.903463 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.903395 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d981410-5bdf-4f71-a44a-7c47ab837672" containerName="maas-api" Jul 09 01:28:13.906531 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.906511 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:13.908957 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.908929 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-88a238-openshift-default-dockercfg-rhnbk\"" Jul 09 01:28:13.917955 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.917933 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h"] Jul 09 01:28:13.931771 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:13.931745 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:14.007010 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.006981 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-certs\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007160 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007014 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-envoy\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007160 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007035 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-podinfo\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007160 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007055 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nxr\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-kube-api-access-k8nxr\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007281 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007164 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-credential-socket\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007281 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007205 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istiod-ca-cert\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007281 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007255 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-socket\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007281 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007279 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-data\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.007433 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.007331 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-token\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.103827 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.103794 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:14.107695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107673 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-podinfo\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107778 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107705 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nxr\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-kube-api-access-k8nxr\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107778 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107737 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-credential-socket\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107778 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107764 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istiod-ca-cert\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107921 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107789 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-socket\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107921 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107805 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-data\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107921 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107850 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-token\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107921 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107880 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-certs\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.107921 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.107904 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-envoy\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.108372 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.108347 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-certs\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.108488 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.108376 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-socket\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.108546 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.108481 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-credential-socket\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.108603 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.108582 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-data\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.108694 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.108675 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istiod-ca-cert\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.110132 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.110107 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-envoy\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.110294 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.110280 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-podinfo\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.115018 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.114997 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nxr\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-kube-api-access-k8nxr\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.115255 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.115236 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-token\") pod \"e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.218412 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.218375 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:14.344090 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:14.344066 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h"] Jul 09 01:28:14.346037 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:28:14.346008 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9cf3db_eb27_48e8_bf84_e453c135cd74.slice/crio-7429c0ebefb209758e426817a8a57a6f458e6893a16bc54a45134188e1cc78b2 WatchSource:0}: Error finding container 7429c0ebefb209758e426817a8a57a6f458e6893a16bc54a45134188e1cc78b2: Status 404 returned error can't find the container with id 7429c0ebefb209758e426817a8a57a6f458e6893a16bc54a45134188e1cc78b2 Jul 09 01:28:15.221881 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:15.221844 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" event={"ID":"0f9cf3db-eb27-48e8-bf84-e453c135cd74","Type":"ContainerStarted","Data":"7429c0ebefb209758e426817a8a57a6f458e6893a16bc54a45134188e1cc78b2"} Jul 09 01:28:18.649710 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:18.649673 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 01:28:18.649966 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:18.649745 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 01:28:18.649966 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:18.649784 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 01:28:19.246046 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:19.246012 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" event={"ID":"0f9cf3db-eb27-48e8-bf84-e453c135cd74","Type":"ContainerStarted","Data":"c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1"} Jul 09 01:28:19.265262 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:19.265206 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" podStartSLOduration=1.9640713939999999 podStartE2EDuration="6.265192843s" totalImagesPullingTime="4.301121449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:28:19.263382221 +0000 UTC m=+1963.541220882" watchObservedRunningTime="2026-07-09 01:28:19.265192843 +0000 UTC m=+1963.543031493" Jul 09 01:28:20.218877 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:20.218845 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:20.223670 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:20.223633 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:20.250994 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:20.250968 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:20.251629 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:20.251611 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:21.161737 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.161698 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:21.231421 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.231388 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:21.239407 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.239380 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq"] Jul 09 01:28:21.242982 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.242966 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.245340 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.245309 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-88a238-serving-cert\"" Jul 09 01:28:21.245340 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.245325 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5rxv2\"" Jul 09 01:28:21.246220 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.246203 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 01:28:21.246325 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.246288 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 01:28:21.251099 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.251078 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq"] Jul 09 01:28:21.299018 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.298991 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:21.370151 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.370120 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-openshift-service-ca\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.370296 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.370254 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jrw\" (UniqueName: \"kubernetes.io/projected/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-kube-api-access-89jrw\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.370341 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.370314 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.471072 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.471048 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jrw\" (UniqueName: \"kubernetes.io/projected/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-kube-api-access-89jrw\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.471209 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.471087 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.471209 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.471124 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-openshift-service-ca\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.471286 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:21.471208 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-serving-cert: secret "maas-api-e2e-shared-a-88a238-serving-cert" not found Jul 09 01:28:21.471286 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:21.471282 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls podName:6a9f6f7a-fbb4-4059-81df-eb98aed3f760 nodeName:}" failed. No retries permitted until 2026-07-09 01:28:21.971262904 +0000 UTC m=+1966.249101541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls") pod "maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" (UID: "6a9f6f7a-fbb4-4059-81df-eb98aed3f760") : secret "maas-api-e2e-shared-a-88a238-serving-cert" not found Jul 09 01:28:21.471783 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.471764 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-openshift-service-ca\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.479484 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.479465 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jrw\" (UniqueName: \"kubernetes.io/projected/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-kube-api-access-89jrw\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.820436 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.820343 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:21.926919 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.926889 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:21.975069 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:21.975039 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:21.975230 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:21.975209 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-serving-cert: secret "maas-api-e2e-shared-a-88a238-serving-cert" not found Jul 09 01:28:21.975321 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:21.975306 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls podName:6a9f6f7a-fbb4-4059-81df-eb98aed3f760 nodeName:}" failed. No retries permitted until 2026-07-09 01:28:22.97528323 +0000 UTC m=+1967.253121862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls") pod "maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" (UID: "6a9f6f7a-fbb4-4059-81df-eb98aed3f760") : secret "maas-api-e2e-shared-a-88a238-serving-cert" not found Jul 09 01:28:22.294336 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:22.294294 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:22.983194 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:22.983161 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:22.985465 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:22.985448 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls\") pod \"maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:23.053840 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:23.053816 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:23.179055 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:23.179029 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq"] Jul 09 01:28:23.180609 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:28:23.180574 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9f6f7a_fbb4_4059_81df_eb98aed3f760.slice/crio-befb2ae703f756abc9ab9a2dfd421dcf62be9e1f98dbddb2ce5624568cd40639 WatchSource:0}: Error finding container befb2ae703f756abc9ab9a2dfd421dcf62be9e1f98dbddb2ce5624568cd40639: Status 404 returned error can't find the container with id befb2ae703f756abc9ab9a2dfd421dcf62be9e1f98dbddb2ce5624568cd40639 Jul 09 01:28:23.265255 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:23.265190 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" event={"ID":"6a9f6f7a-fbb4-4059-81df-eb98aed3f760","Type":"ContainerStarted","Data":"befb2ae703f756abc9ab9a2dfd421dcf62be9e1f98dbddb2ce5624568cd40639"} Jul 09 01:28:24.270868 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:24.270833 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" event={"ID":"6a9f6f7a-fbb4-4059-81df-eb98aed3f760","Type":"ContainerStarted","Data":"e2bfbb37dd1a8fc70ac39ddcea166ed86cca0a170cdb6ca74186dda2a9eec8eb"} Jul 09 01:28:24.271303 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:24.270986 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:24.287812 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:24.287771 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" podStartSLOduration=2.969419903 podStartE2EDuration="3.287758438s" totalImagesPullingTime="318.338535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:28:24.286424025 +0000 UTC m=+1968.564262677" watchObservedRunningTime="2026-07-09 01:28:24.287758438 +0000 UTC m=+1968.565597089" Jul 09 01:28:25.675409 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:25.675377 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:25.997933 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:25.997897 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:26.293611 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:26.293533 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:26.811744 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:26.811714 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:29.114574 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.114494 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:29.189552 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.189526 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2"] Jul 09 01:28:29.193096 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.193076 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.193971 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.193946 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:29.195438 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.195421 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-52d242-serving-cert\"" Jul 09 01:28:29.202383 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.202363 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2"] Jul 09 01:28:29.237478 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.237453 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmzn\" (UniqueName: \"kubernetes.io/projected/18423ae6-5362-4711-9d97-f88086bb7f67-kube-api-access-ttmzn\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.237567 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.237514 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18423ae6-5362-4711-9d97-f88086bb7f67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.237567 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.237543 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.281425 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.281399 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:29.338102 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.338071 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.338230 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:29.338211 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-serving-cert: secret "maas-api-e2e-shared-b-52d242-serving-cert" not found Jul 09 01:28:29.338268 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.338227 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmzn\" (UniqueName: \"kubernetes.io/projected/18423ae6-5362-4711-9d97-f88086bb7f67-kube-api-access-ttmzn\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.338316 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:28:29.338289 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls podName:18423ae6-5362-4711-9d97-f88086bb7f67 nodeName:}" failed. No retries permitted until 2026-07-09 01:28:29.838267515 +0000 UTC m=+1974.116106161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls") pod "maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" (UID: "18423ae6-5362-4711-9d97-f88086bb7f67") : secret "maas-api-e2e-shared-b-52d242-serving-cert" not found Jul 09 01:28:29.338359 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.338348 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18423ae6-5362-4711-9d97-f88086bb7f67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.338988 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.338971 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18423ae6-5362-4711-9d97-f88086bb7f67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.351424 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.351402 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmzn\" (UniqueName: \"kubernetes.io/projected/18423ae6-5362-4711-9d97-f88086bb7f67-kube-api-access-ttmzn\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.746763 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.746726 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:29.842740 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.842711 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.845047 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.845014 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls\") pod \"maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:29.852209 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:29.852171 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:30.105328 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:30.105243 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:30.250665 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:30.250624 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2"] Jul 09 01:28:30.253154 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:28:30.253130 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18423ae6_5362_4711_9d97_f88086bb7f67.slice/crio-1e56558f3e903243c8679dd9a3caa35a0952564ca1245393424299a62cba31f0 WatchSource:0}: Error finding container 1e56558f3e903243c8679dd9a3caa35a0952564ca1245393424299a62cba31f0: Status 404 returned error can't find the container with id 1e56558f3e903243c8679dd9a3caa35a0952564ca1245393424299a62cba31f0 Jul 09 01:28:30.298358 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:30.298325 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:28:30.299041 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:30.299012 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" event={"ID":"18423ae6-5362-4711-9d97-f88086bb7f67","Type":"ContainerStarted","Data":"1e56558f3e903243c8679dd9a3caa35a0952564ca1245393424299a62cba31f0"} Jul 09 01:28:31.304565 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:31.304527 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" event={"ID":"18423ae6-5362-4711-9d97-f88086bb7f67","Type":"ContainerStarted","Data":"404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325"} Jul 09 01:28:31.305040 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:31.304701 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:31.321809 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:31.321767 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" podStartSLOduration=1.9191418470000001 podStartE2EDuration="2.321754168s" totalImagesPullingTime="402.612321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:28:31.320016035 +0000 UTC m=+1975.597854687" watchObservedRunningTime="2026-07-09 01:28:31.321754168 +0000 UTC m=+1975.599592818" Jul 09 01:28:33.616073 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:33.616043 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:36.291779 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:36.291747 2543 patch_prober.go:28] interesting pod/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 01:28:36.292147 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:36.291831 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 01:28:36.342279 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:36.342252 2543 scope.go:117] "RemoveContainer" containerID="40bda183adc5c2a02f3082177606613fea9f3ea88dfde18065a309a02ea7858f" Jul 09 01:28:37.314449 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:37.314380 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:28:37.334155 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:37.334128 2543 generic.go:356] "Generic (PLEG): container finished" podID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerID="c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1" exitCode=0 Jul 09 01:28:37.334285 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:37.334177 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" event={"ID":"0f9cf3db-eb27-48e8-bf84-e453c135cd74","Type":"ContainerDied","Data":"c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1"} Jul 09 01:28:37.334542 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:37.334526 2543 scope.go:117] "RemoveContainer" containerID="c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1" Jul 09 01:28:37.335563 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:37.335538 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 01:28:37.335627 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:37.335612 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 01:28:37.335689 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:37.335664 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 01:28:38.340170 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:38.340129 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" event={"ID":"0f9cf3db-eb27-48e8-bf84-e453c135cd74","Type":"ContainerStarted","Data":"2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b"} Jul 09 01:28:39.218922 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:39.218896 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:39.219149 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:39.219125 2543 patch_prober.go:28] interesting pod/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 09 01:28:39.219215 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:39.219194 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 09 01:28:40.219617 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:40.219583 2543 patch_prober.go:28] interesting pod/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 09 01:28:40.220038 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:40.219636 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 09 01:28:41.219787 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:41.219751 2543 patch_prober.go:28] interesting pod/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 09 01:28:41.220180 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:41.219815 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 09 01:28:42.222246 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:42.222219 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:42.222617 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:42.222573 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:42.223322 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:42.223306 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:28:43.338409 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:43.338371 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:43.436830 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:43.436788 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:50.590889 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:50.590831 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:50.697037 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:50.697000 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:28:56.360985 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:28:56.360931 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:29:08.153191 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:29:08.153148 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:29:08.254914 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:29:08.254875 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:29:23.159767 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:29:23.159732 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:30:00.133210 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.133132 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc"] Jul 09 01:30:00.136285 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.136267 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" Jul 09 01:30:00.144021 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.143998 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc"] Jul 09 01:30:00.288365 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.288335 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45d5\" (UniqueName: \"kubernetes.io/projected/87901140-5480-4f1a-9a56-3b4c59b3d954-kube-api-access-w45d5\") pod \"maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc\" (UID: \"87901140-5480-4f1a-9a56-3b4c59b3d954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" Jul 09 01:30:00.388922 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.388846 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45d5\" (UniqueName: \"kubernetes.io/projected/87901140-5480-4f1a-9a56-3b4c59b3d954-kube-api-access-w45d5\") pod \"maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc\" (UID: \"87901140-5480-4f1a-9a56-3b4c59b3d954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" Jul 09 01:30:00.396473 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.396445 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45d5\" (UniqueName: \"kubernetes.io/projected/87901140-5480-4f1a-9a56-3b4c59b3d954-kube-api-access-w45d5\") pod \"maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc\" (UID: \"87901140-5480-4f1a-9a56-3b4c59b3d954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" Jul 09 01:30:00.447465 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.447440 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" Jul 09 01:30:00.578237 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.578209 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc"] Jul 09 01:30:00.580131 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:30:00.580112 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87901140_5480_4f1a_9a56_3b4c59b3d954.slice/crio-6c719898359a722b46b86caa076454b7322eed0e8b612ce87ef91d3b3199e348 WatchSource:0}: Error finding container 6c719898359a722b46b86caa076454b7322eed0e8b612ce87ef91d3b3199e348: Status 404 returned error can't find the container with id 6c719898359a722b46b86caa076454b7322eed0e8b612ce87ef91d3b3199e348 Jul 09 01:30:00.686954 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:00.686877 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" event={"ID":"87901140-5480-4f1a-9a56-3b4c59b3d954","Type":"ContainerStarted","Data":"6c719898359a722b46b86caa076454b7322eed0e8b612ce87ef91d3b3199e348"} Jul 09 01:30:01.693059 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:01.693022 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" event={"ID":"87901140-5480-4f1a-9a56-3b4c59b3d954","Type":"ContainerStarted","Data":"968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260"} Jul 09 01:30:01.708361 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:01.708313 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" podStartSLOduration=1.708299051 podStartE2EDuration="1.708299051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:30:01.706583162 +0000 UTC m=+2065.984421796" watchObservedRunningTime="2026-07-09 01:30:01.708299051 +0000 UTC m=+2065.986137745" Jul 09 01:30:04.632723 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:04.632687 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:30:04.685356 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:30:04.685324 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:31:00.015725 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:00.015686 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc"] Jul 09 01:31:00.016703 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:00.016626 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" podUID="87901140-5480-4f1a-9a56-3b4c59b3d954" containerName="cleanup" containerID="cri-o://968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260" gracePeriod=30 Jul 09 01:31:30.203527 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:30.203501 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" Jul 09 01:31:30.285223 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:30.285164 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w45d5\" (UniqueName: \"kubernetes.io/projected/87901140-5480-4f1a-9a56-3b4c59b3d954-kube-api-access-w45d5\") pod \"87901140-5480-4f1a-9a56-3b4c59b3d954\" (UID: \"87901140-5480-4f1a-9a56-3b4c59b3d954\") " Jul 09 01:31:30.287134 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:30.287113 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87901140-5480-4f1a-9a56-3b4c59b3d954-kube-api-access-w45d5" (OuterVolumeSpecName: "kube-api-access-w45d5") pod "87901140-5480-4f1a-9a56-3b4c59b3d954" (UID: "87901140-5480-4f1a-9a56-3b4c59b3d954"). InnerVolumeSpecName "kube-api-access-w45d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:31:30.386167 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:30.386144 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45d5\" (UniqueName: \"kubernetes.io/projected/87901140-5480-4f1a-9a56-3b4c59b3d954-kube-api-access-w45d5\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:31:31.072099 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.072062 2543 generic.go:356] "Generic (PLEG): container finished" podID="87901140-5480-4f1a-9a56-3b4c59b3d954" containerID="968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260" exitCode=137 Jul 09 01:31:31.072279 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.072116 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" Jul 09 01:31:31.072279 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.072143 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" event={"ID":"87901140-5480-4f1a-9a56-3b4c59b3d954","Type":"ContainerDied","Data":"968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260"} Jul 09 01:31:31.072279 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.072203 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc" event={"ID":"87901140-5480-4f1a-9a56-3b4c59b3d954","Type":"ContainerDied","Data":"6c719898359a722b46b86caa076454b7322eed0e8b612ce87ef91d3b3199e348"} Jul 09 01:31:31.072279 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.072236 2543 scope.go:117] "RemoveContainer" containerID="968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260" Jul 09 01:31:31.079996 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.079986 2543 scope.go:117] "RemoveContainer" containerID="968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260" Jul 09 01:31:31.080241 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:31:31.080223 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260\": container with ID starting with 968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260 not found: ID does not exist" containerID="968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260" Jul 09 01:31:31.080286 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.080248 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260"} err="failed to get container status \"968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260\": rpc error: code = NotFound desc = could not find container \"968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260\": container with ID starting with 968b6f6d35e1783b477f78c6a473538c5643a09d56d047ac8db4786dbf28f260 not found: ID does not exist" Jul 09 01:31:31.087245 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.087216 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc"] Jul 09 01:31:31.089449 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:31.089431 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-52d242-29726010-4xccc"] Jul 09 01:31:32.928157 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:32.928104 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87901140-5480-4f1a-9a56-3b4c59b3d954" path="/var/lib/kubelet/pods/87901140-5480-4f1a-9a56-3b4c59b3d954/volumes" Jul 09 01:31:42.410557 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:42.410523 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:31:43.158228 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:43.158177 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:31:45.386831 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:45.386800 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:31:48.067817 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:31:48.067777 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:32:18.952998 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:32:18.952953 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:32:49.189713 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:32:49.189675 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:32:50.340942 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:32:50.340903 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:05.035292 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:05.035218 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:05.733167 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:05.733131 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:16.955220 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:16.955188 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:37.140185 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:37.140063 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:44.034188 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:44.034146 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:44.131332 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:44.131295 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:51.283624 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:51.283589 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:33:51.384156 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:33:51.384122 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:28.062950 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:28.062916 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:28.963383 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:28.963342 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:47.980227 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:47.980187 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:48.387744 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:48.387629 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:48.487912 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:48.487882 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:48.600734 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:48.600699 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq"] Jul 09 01:34:48.600952 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:48.600929 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" podUID="6a9f6f7a-fbb4-4059-81df-eb98aed3f760" containerName="maas-api" containerID="cri-o://e2bfbb37dd1a8fc70ac39ddcea166ed86cca0a170cdb6ca74186dda2a9eec8eb" gracePeriod=30 Jul 09 01:34:48.640477 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:48.640402 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:48.928837 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:48.928806 2543 generic.go:356] "Generic (PLEG): container finished" podID="6a9f6f7a-fbb4-4059-81df-eb98aed3f760" containerID="e2bfbb37dd1a8fc70ac39ddcea166ed86cca0a170cdb6ca74186dda2a9eec8eb" exitCode=0 Jul 09 01:34:48.929754 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:48.929732 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" event={"ID":"6a9f6f7a-fbb4-4059-81df-eb98aed3f760","Type":"ContainerDied","Data":"e2bfbb37dd1a8fc70ac39ddcea166ed86cca0a170cdb6ca74186dda2a9eec8eb"} Jul 09 01:34:49.041316 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.041288 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:34:49.159155 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.159128 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-openshift-service-ca\") pod \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " Jul 09 01:34:49.159300 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.159170 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89jrw\" (UniqueName: \"kubernetes.io/projected/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-kube-api-access-89jrw\") pod \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " Jul 09 01:34:49.159300 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.159187 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls\") pod \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\" (UID: \"6a9f6f7a-fbb4-4059-81df-eb98aed3f760\") " Jul 09 01:34:49.159457 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.159434 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a9f6f7a-fbb4-4059-81df-eb98aed3f760" (UID: "6a9f6f7a-fbb4-4059-81df-eb98aed3f760"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:34:49.161100 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.161079 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a9f6f7a-fbb4-4059-81df-eb98aed3f760" (UID: "6a9f6f7a-fbb4-4059-81df-eb98aed3f760"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:34:49.161264 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.161249 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-kube-api-access-89jrw" (OuterVolumeSpecName: "kube-api-access-89jrw") pod "6a9f6f7a-fbb4-4059-81df-eb98aed3f760" (UID: "6a9f6f7a-fbb4-4059-81df-eb98aed3f760"). InnerVolumeSpecName "kube-api-access-89jrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:34:49.260404 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.260381 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:49.260404 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.260402 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89jrw\" (UniqueName: \"kubernetes.io/projected/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-kube-api-access-89jrw\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:49.260530 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.260412 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9f6f7a-fbb4-4059-81df-eb98aed3f760-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:49.934226 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.934198 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" Jul 09 01:34:49.934393 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.934201 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq" event={"ID":"6a9f6f7a-fbb4-4059-81df-eb98aed3f760","Type":"ContainerDied","Data":"befb2ae703f756abc9ab9a2dfd421dcf62be9e1f98dbddb2ce5624568cd40639"} Jul 09 01:34:49.934393 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.934338 2543 scope.go:117] "RemoveContainer" containerID="e2bfbb37dd1a8fc70ac39ddcea166ed86cca0a170cdb6ca74186dda2a9eec8eb" Jul 09 01:34:49.953824 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.953801 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq"] Jul 09 01:34:49.957279 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:49.957254 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88a238-57f96d7dfd-hz7pq"] Jul 09 01:34:50.928532 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:50.928499 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9f6f7a-fbb4-4059-81df-eb98aed3f760" path="/var/lib/kubelet/pods/6a9f6f7a-fbb4-4059-81df-eb98aed3f760/volumes" Jul 09 01:34:54.293633 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:54.293596 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:54.350749 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:54.350719 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:54.534104 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:54.534066 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h"] Jul 09 01:34:54.534366 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:54.534309 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" containerID="cri-o://2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b" gracePeriod=30 Jul 09 01:34:54.553606 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:54.553535 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:54.951833 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:54.951799 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:55.373311 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:55.373227 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:55.597911 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:55.597880 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:55.698824 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:55.698789 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:55.822809 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:55.822767 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2"] Jul 09 01:34:55.823045 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:55.823019 2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" podUID="18423ae6-5362-4711-9d97-f88086bb7f67" containerName="maas-api" containerID="cri-o://404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325" gracePeriod=30 Jul 09 01:34:55.851733 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:55.851703 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:34:56.262476 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.262451 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:34:56.421315 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.421233 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttmzn\" (UniqueName: \"kubernetes.io/projected/18423ae6-5362-4711-9d97-f88086bb7f67-kube-api-access-ttmzn\") pod \"18423ae6-5362-4711-9d97-f88086bb7f67\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " Jul 09 01:34:56.421315 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.421308 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18423ae6-5362-4711-9d97-f88086bb7f67-openshift-service-ca\") pod \"18423ae6-5362-4711-9d97-f88086bb7f67\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " Jul 09 01:34:56.421741 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.421357 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls\") pod \"18423ae6-5362-4711-9d97-f88086bb7f67\" (UID: \"18423ae6-5362-4711-9d97-f88086bb7f67\") " Jul 09 01:34:56.421741 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.421697 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18423ae6-5362-4711-9d97-f88086bb7f67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18423ae6-5362-4711-9d97-f88086bb7f67" (UID: "18423ae6-5362-4711-9d97-f88086bb7f67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:34:56.423418 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.423390 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18423ae6-5362-4711-9d97-f88086bb7f67-kube-api-access-ttmzn" (OuterVolumeSpecName: "kube-api-access-ttmzn") pod "18423ae6-5362-4711-9d97-f88086bb7f67" (UID: "18423ae6-5362-4711-9d97-f88086bb7f67"). InnerVolumeSpecName "kube-api-access-ttmzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:34:56.423418 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.423405 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18423ae6-5362-4711-9d97-f88086bb7f67" (UID: "18423ae6-5362-4711-9d97-f88086bb7f67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 01:34:56.522147 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.522116 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttmzn\" (UniqueName: \"kubernetes.io/projected/18423ae6-5362-4711-9d97-f88086bb7f67-kube-api-access-ttmzn\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:56.522147 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.522144 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18423ae6-5362-4711-9d97-f88086bb7f67-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:56.522331 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.522172 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18423ae6-5362-4711-9d97-f88086bb7f67-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:56.963764 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.963731 2543 generic.go:356] "Generic (PLEG): container finished" podID="18423ae6-5362-4711-9d97-f88086bb7f67" containerID="404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325" exitCode=0 Jul 09 01:34:56.963888 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.963770 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" Jul 09 01:34:56.963888 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.963781 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" event={"ID":"18423ae6-5362-4711-9d97-f88086bb7f67","Type":"ContainerDied","Data":"404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325"} Jul 09 01:34:56.963888 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.963815 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2" event={"ID":"18423ae6-5362-4711-9d97-f88086bb7f67","Type":"ContainerDied","Data":"1e56558f3e903243c8679dd9a3caa35a0952564ca1245393424299a62cba31f0"} Jul 09 01:34:56.963888 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.963835 2543 scope.go:117] "RemoveContainer" containerID="404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325" Jul 09 01:34:56.971816 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.971800 2543 scope.go:117] "RemoveContainer" containerID="404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325" Jul 09 01:34:56.972069 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:34:56.972046 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325\": container with ID starting with 404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325 not found: ID does not exist" containerID="404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325" Jul 09 01:34:56.972137 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.972079 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325"} err="failed to get container status \"404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325\": rpc error: code = NotFound desc = could not find container \"404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325\": container with ID starting with 404cdbe457cea18b2be27f543eb897bf2105aafef4c57c782461b152f7e7b325 not found: ID does not exist" Jul 09 01:34:56.978754 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.978730 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2"] Jul 09 01:34:56.982008 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:56.981974 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-52d242-5c676bf798-2tpc2"] Jul 09 01:34:57.224405 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:57.224377 2543 patch_prober.go:28] interesting pod/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 01:34:57.224583 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:57.224435 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 01:34:58.928490 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:58.928461 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18423ae6-5362-4711-9d97-f88086bb7f67" path="/var/lib/kubelet/pods/18423ae6-5362-4711-9d97-f88086bb7f67/volumes" Jul 09 01:34:59.779933 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.779910 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:34:59.848732 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848660 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-credential-socket\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.848732 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848694 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-socket\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.848732 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848719 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-envoy\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.848987 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848763 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8nxr\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-kube-api-access-k8nxr\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.848987 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848781 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-podinfo\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.848987 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848803 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-token\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.848987 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848841 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-data\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.848987 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.848914 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 01:34:59.849248 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849003 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 01:34:59.849248 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849029 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istiod-ca-cert\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.849248 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849071 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-certs\") pod \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\" (UID: \"0f9cf3db-eb27-48e8-bf84-e453c135cd74\") " Jul 09 01:34:59.849403 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849340 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 01:34:59.849403 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849382 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.849509 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849403 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.849567 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849513 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 01:34:59.849624 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.849571 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 01:34:59.851092 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.851064 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 01:34:59.851213 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.851111 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 01:34:59.851294 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.851274 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-kube-api-access-k8nxr" (OuterVolumeSpecName: "kube-api-access-k8nxr") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "kube-api-access-k8nxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:34:59.851433 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.851413 2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f9cf3db-eb27-48e8-bf84-e453c135cd74" (UID: "0f9cf3db-eb27-48e8-bf84-e453c135cd74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 01:34:59.950748 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.950717 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.950748 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.950744 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.951100 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.950757 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.951100 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.950769 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.951100 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.950780 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8nxr\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-kube-api-access-k8nxr\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.951100 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.950793 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.951100 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.950805 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9cf3db-eb27-48e8-bf84-e453c135cd74-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Jul 09 01:34:59.978186 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.978154 2543 generic.go:356] "Generic (PLEG): container finished" podID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerID="2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b" exitCode=0 Jul 09 01:34:59.978293 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.978209 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" Jul 09 01:34:59.978293 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.978242 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" event={"ID":"0f9cf3db-eb27-48e8-bf84-e453c135cd74","Type":"ContainerDied","Data":"2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b"} Jul 09 01:34:59.978363 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.978303 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h" event={"ID":"0f9cf3db-eb27-48e8-bf84-e453c135cd74","Type":"ContainerDied","Data":"7429c0ebefb209758e426817a8a57a6f458e6893a16bc54a45134188e1cc78b2"} Jul 09 01:34:59.978363 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.978326 2543 scope.go:117] "RemoveContainer" containerID="2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b" Jul 09 01:34:59.987085 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.987069 2543 scope.go:117] "RemoveContainer" containerID="c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1" Jul 09 01:34:59.994083 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.994065 2543 scope.go:117] "RemoveContainer" containerID="2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b" Jul 09 01:34:59.994321 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:34:59.994304 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b\": container with ID starting with 2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b not found: ID does not exist" containerID="2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b" Jul 09 01:34:59.994368 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.994327 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b"} err="failed to get container status \"2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b\": rpc error: code = NotFound desc = could not find container \"2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b\": container with ID starting with 2540930a4b038740ba6caef1da292e6a0b2df16510994d20b2d2291eb14bc83b not found: ID does not exist" Jul 09 01:34:59.994368 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.994343 2543 scope.go:117] "RemoveContainer" containerID="c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1" Jul 09 01:34:59.994556 ip-10-0-134-128 kubenswrapper[2543]: E0709 01:34:59.994532 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1\": container with ID starting with c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1 not found: ID does not exist" containerID="c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1" Jul 09 01:34:59.994611 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.994567 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1"} err="failed to get container status \"c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1\": rpc error: code = NotFound desc = could not find container \"c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1\": container with ID starting with c46166801831918cf56926e55db64c8c121982c7e471612edbf02d7c3b8745c1 not found: ID does not exist" Jul 09 01:34:59.997894 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:34:59.997873 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h"] Jul 09 01:35:00.001714 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:00.001695 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-88a238-openshift-default-6f5f9c75d7-98f5h"] Jul 09 01:35:00.927589 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:00.927558 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" path="/var/lib/kubelet/pods/0f9cf3db-eb27-48e8-bf84-e453c135cd74/volumes" Jul 09 01:35:01.314324 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:01.314290 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:35:01.414263 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:01.414237 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:35:01.574508 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:01.574412 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:35:07.616445 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:07.616416 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85699f56b9-gxsq5_135d3237-3b4d-44a9-b438-831ad0d66d43/copy-manifests/0.log" Jul 09 01:35:07.622932 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:07.622896 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85699f56b9-gxsq5_135d3237-3b4d-44a9-b438-831ad0d66d43/manager/0.log" Jul 09 01:35:10.921917 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:10.921888 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w7jdz_036c5583-b903-4473-8c94-8dfb6fa73b91/discovery/0.log" Jul 09 01:35:11.033133 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:11.033107 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7556c75c9b-7tw9t_5b596c63-6944-4722-ace5-8978ff269447/kube-auth-proxy/0.log" Jul 09 01:35:16.429281 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:16.429248 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:35:16.493214 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:16.493170 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:35:19.269804 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:19.269775 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-thjzt_b581286c-01e6-45a0-a760-5bb689a8b8e3/global-pull-secret-syncer/0.log" Jul 09 01:35:19.313840 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:19.313818 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" Jul 09 01:35:19.441435 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:19.441408 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-128.ec2.internal_1c46fe0b45721f50ed4b3952298077b2/haproxy/0.log" Jul 09 01:35:26.883588 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:26.883560 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tzsdg_a6ceb525-bbe6-4c26-b8eb-cf1505989264/kube-state-metrics/0.log" Jul 09 01:35:26.900045 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:26.899957 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tzsdg_a6ceb525-bbe6-4c26-b8eb-cf1505989264/kube-rbac-proxy-main/0.log" Jul 09 01:35:26.920560 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:26.920544 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tzsdg_a6ceb525-bbe6-4c26-b8eb-cf1505989264/kube-rbac-proxy-self/0.log" Jul 09 01:35:26.975538 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:26.975516 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-w27gw_cf817aca-fa10-4b4f-967c-979c65d5bd64/monitoring-plugin/0.log" Jul 09 01:35:27.159232 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.159176 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-knngl_7b21af0f-9034-4a91-b7c2-27b34cce3685/node-exporter/0.log" Jul 09 01:35:27.178306 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.178287 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-knngl_7b21af0f-9034-4a91-b7c2-27b34cce3685/kube-rbac-proxy/0.log" Jul 09 01:35:27.197682 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.197664 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-knngl_7b21af0f-9034-4a91-b7c2-27b34cce3685/init-textfile/0.log" Jul 09 01:35:27.306749 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.306728 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d598fcf-d7bb-4a28-a559-c726182008ea/prometheus/0.log" Jul 09 01:35:27.335759 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.335742 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d598fcf-d7bb-4a28-a559-c726182008ea/config-reloader/0.log" Jul 09 01:35:27.354574 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.354553 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d598fcf-d7bb-4a28-a559-c726182008ea/thanos-sidecar/0.log" Jul 09 01:35:27.373112 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.373094 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d598fcf-d7bb-4a28-a559-c726182008ea/kube-rbac-proxy-web/0.log" Jul 09 01:35:27.394767 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.394742 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d598fcf-d7bb-4a28-a559-c726182008ea/kube-rbac-proxy/0.log" Jul 09 01:35:27.411855 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.411809 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d598fcf-d7bb-4a28-a559-c726182008ea/kube-rbac-proxy-thanos/0.log" Jul 09 01:35:27.431637 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.431620 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d598fcf-d7bb-4a28-a559-c726182008ea/init-config-reloader/0.log" Jul 09 01:35:27.459209 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.459172 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qhd7c_2a74b308-8be9-4614-866f-1629491baed0/prometheus-operator/0.log" Jul 09 01:35:27.477682 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.477666 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qhd7c_2a74b308-8be9-4614-866f-1629491baed0/kube-rbac-proxy/0.log" Jul 09 01:35:27.591607 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.591585 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bf8455844-bsdkw_f84cb37f-3c84-45fe-bca4-61b562dcec18/thanos-query/0.log" Jul 09 01:35:27.610374 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.610357 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bf8455844-bsdkw_f84cb37f-3c84-45fe-bca4-61b562dcec18/kube-rbac-proxy-web/0.log" Jul 09 01:35:27.629597 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.629581 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bf8455844-bsdkw_f84cb37f-3c84-45fe-bca4-61b562dcec18/kube-rbac-proxy/0.log" Jul 09 01:35:27.649932 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.649911 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bf8455844-bsdkw_f84cb37f-3c84-45fe-bca4-61b562dcec18/prom-label-proxy/0.log" Jul 09 01:35:27.671111 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.671028 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bf8455844-bsdkw_f84cb37f-3c84-45fe-bca4-61b562dcec18/kube-rbac-proxy-rules/0.log" Jul 09 01:35:27.692506 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:27.692488 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bf8455844-bsdkw_f84cb37f-3c84-45fe-bca4-61b562dcec18/kube-rbac-proxy-metrics/0.log" Jul 09 01:35:29.449277 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449242 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk"] Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449545 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449556 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449564 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9f6f7a-fbb4-4059-81df-eb98aed3f760" containerName="maas-api" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449570 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9f6f7a-fbb4-4059-81df-eb98aed3f760" containerName="maas-api" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449581 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87901140-5480-4f1a-9a56-3b4c59b3d954" containerName="cleanup" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449587 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="87901140-5480-4f1a-9a56-3b4c59b3d954" containerName="cleanup" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449595 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18423ae6-5362-4711-9d97-f88086bb7f67" containerName="maas-api" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449600 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="18423ae6-5362-4711-9d97-f88086bb7f67" containerName="maas-api" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449673 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9f6f7a-fbb4-4059-81df-eb98aed3f760" containerName="maas-api" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449682 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449688 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9cf3db-eb27-48e8-bf84-e453c135cd74" containerName="istio-proxy" Jul 09 01:35:29.449690 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449694 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="18423ae6-5362-4711-9d97-f88086bb7f67" containerName="maas-api" Jul 09 01:35:29.450111 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.449701 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="87901140-5480-4f1a-9a56-3b4c59b3d954" containerName="cleanup" Jul 09 01:35:29.452802 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.452787 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.454943 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.454922 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6bs7j\"/\"kube-root-ca.crt\"" Jul 09 01:35:29.455863 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.455845 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6bs7j\"/\"default-dockercfg-r8h45\"" Jul 09 01:35:29.455960 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.455847 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6bs7j\"/\"openshift-service-ca.crt\"" Jul 09 01:35:29.458787 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.458760 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk"] Jul 09 01:35:29.477793 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.477769 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-proc\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.477892 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.477796 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-sys\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.477892 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.477813 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-lib-modules\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.477892 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.477852 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxrt\" (UniqueName: \"kubernetes.io/projected/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-kube-api-access-pmxrt\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.477892 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.477875 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-podres\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578561 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578535 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-proc\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578566 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-sys\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578591 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-lib-modules\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578626 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxrt\" (UniqueName: \"kubernetes.io/projected/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-kube-api-access-pmxrt\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578669 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-podres\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578668 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-sys\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578668 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-proc\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578927 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578769 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-podres\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.578927 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.578775 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-lib-modules\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.585777 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.585749 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxrt\" (UniqueName: \"kubernetes.io/projected/3c75c9f9-bb74-471c-acc1-6ceb23f171b9-kube-api-access-pmxrt\") pod \"perf-node-gather-daemonset-c75bk\" (UID: \"3c75c9f9-bb74-471c-acc1-6ceb23f171b9\") " pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.764289 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.764214 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:29.887193 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:29.887158 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk"] Jul 09 01:35:29.889611 ip-10-0-134-128 kubenswrapper[2543]: W0709 01:35:29.889581 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3c75c9f9_bb74_471c_acc1_6ceb23f171b9.slice/crio-5d4ba4e80d5275220b5805bc6398185edcfc54540651c563ca2e5e963b67fade WatchSource:0}: Error finding container 5d4ba4e80d5275220b5805bc6398185edcfc54540651c563ca2e5e963b67fade: Status 404 returned error can't find the container with id 5d4ba4e80d5275220b5805bc6398185edcfc54540651c563ca2e5e963b67fade Jul 09 01:35:30.101056 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:30.100975 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" event={"ID":"3c75c9f9-bb74-471c-acc1-6ceb23f171b9","Type":"ContainerStarted","Data":"14d97739664bf20878b23ea2511e0f95c062b98c4d7b2366f69e7992fb3cec64"} Jul 09 01:35:30.101056 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:30.101021 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" event={"ID":"3c75c9f9-bb74-471c-acc1-6ceb23f171b9","Type":"ContainerStarted","Data":"5d4ba4e80d5275220b5805bc6398185edcfc54540651c563ca2e5e963b67fade"} Jul 09 01:35:30.101251 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:30.101104 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:30.115029 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:30.114976 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" podStartSLOduration=1.114959815 podStartE2EDuration="1.114959815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:35:30.114780866 +0000 UTC m=+2394.392619544" watchObservedRunningTime="2026-07-09 01:35:30.114959815 +0000 UTC m=+2394.392798458" Jul 09 01:35:31.715562 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:31.715532 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h6zd8_b04a770f-3b16-42aa-8524-42e84978b874/dns/0.log" Jul 09 01:35:31.736221 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:31.736195 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h6zd8_b04a770f-3b16-42aa-8524-42e84978b874/kube-rbac-proxy/0.log" Jul 09 01:35:31.820900 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:31.820876 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klfjr_b53d14f6-bfe9-4d19-ad46-d16aea78249c/dns-node-resolver/0.log" Jul 09 01:35:32.401352 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:32.401325 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s55gv_0b9f359d-cb28-4cf9-90ce-248aa8fbd878/node-ca/0.log" Jul 09 01:35:33.422923 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:33.422884 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w7jdz_036c5583-b903-4473-8c94-8dfb6fa73b91/discovery/0.log" Jul 09 01:35:33.442947 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:33.442921 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7556c75c9b-7tw9t_5b596c63-6944-4722-ace5-8978ff269447/kube-auth-proxy/0.log" Jul 09 01:35:34.408626 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:34.408595 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7h8np_29569db2-280a-4e93-a06e-0400f7ba321a/serve-healthcheck-canary/0.log" Jul 09 01:35:34.966266 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:34.966236 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzg2n_56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee/kube-rbac-proxy/0.log" Jul 09 01:35:34.984703 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:34.984684 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzg2n_56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee/exporter/0.log" Jul 09 01:35:35.003875 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:35.003854 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzg2n_56b3d4c8-2965-4e45-ba5a-2ae523a4b6ee/extractor/0.log" Jul 09 01:35:36.114336 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:36.114312 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6bs7j/perf-node-gather-daemonset-c75bk" Jul 09 01:35:37.678695 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:37.678633 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85699f56b9-gxsq5_135d3237-3b4d-44a9-b438-831ad0d66d43/manager/0.log" Jul 09 01:35:37.697455 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:37.697432 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85699f56b9-gxsq5_135d3237-3b4d-44a9-b438-831ad0d66d43/copy-manifests/0.log" Jul 09 01:35:39.296014 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:39.295994 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-75764c764c-fn844_cf26093c-ffc1-486d-866d-014facd70a06/manager/0.log" Jul 09 01:35:39.350134 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:39.350106 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-x6zwv_e367f5d8-b32d-46ec-8202-72db8ca2323d/openshift-lws-operator/0.log" Jul 09 01:35:40.054420 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:40.054389 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w7jdz_036c5583-b903-4473-8c94-8dfb6fa73b91/discovery/0.log" Jul 09 01:35:40.066608 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:40.066583 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7556c75c9b-7tw9t_5b596c63-6944-4722-ace5-8978ff269447/kube-auth-proxy/0.log" Jul 09 01:35:41.681635 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:41.681606 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w7jdz_036c5583-b903-4473-8c94-8dfb6fa73b91/discovery/0.log" Jul 09 01:35:41.701550 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:41.701530 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7556c75c9b-7tw9t_5b596c63-6944-4722-ace5-8978ff269447/kube-auth-proxy/0.log" Jul 09 01:35:45.734492 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:45.734467 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9qmvg_8afaa9e7-52cc-4050-9a84-03208d9d39c2/kube-multus/0.log" Jul 09 01:35:46.039525 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.039453 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjb6p_2a0c7205-318e-465c-bd0f-1e4734d2b8e5/kube-multus-additional-cni-plugins/0.log" Jul 09 01:35:46.061290 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.061271 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjb6p_2a0c7205-318e-465c-bd0f-1e4734d2b8e5/egress-router-binary-copy/0.log" Jul 09 01:35:46.081076 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.081057 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjb6p_2a0c7205-318e-465c-bd0f-1e4734d2b8e5/cni-plugins/0.log" Jul 09 01:35:46.101401 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.101383 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjb6p_2a0c7205-318e-465c-bd0f-1e4734d2b8e5/bond-cni-plugin/0.log" Jul 09 01:35:46.122266 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.122252 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjb6p_2a0c7205-318e-465c-bd0f-1e4734d2b8e5/routeoverride-cni/0.log" Jul 09 01:35:46.139468 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.139447 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-649tx_c95d9938-3b58-44fd-a335-b743e9772e5e/konnectivity-agent/0.log" limit=1024 Jul 09 01:35:46.142219 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.142203 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjb6p_2a0c7205-318e-465c-bd0f-1e4734d2b8e5/whereabouts-cni-bincopy/0.log" Jul 09 01:35:46.181100 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.181079 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjb6p_2a0c7205-318e-465c-bd0f-1e4734d2b8e5/whereabouts-cni/0.log" Jul 09 01:35:46.252917 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.252901 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-69lnl_c541c940-d9b8-4cb0-ac38-5abdc3e112c8/network-metrics-daemon/0.log" Jul 09 01:35:46.269918 ip-10-0-134-128 kubenswrapper[2543]: I0709 01:35:46.269903 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-69lnl_c541c940-d9b8-4cb0-ac38-5abdc3e112c8/kube-rbac-proxy/0.log"